givenscj, Author at ACS - Page 15 of 43

August 26, 2013

Programmatically working with Device Channels

There are several questions about how to create and assign device channels programmatically. I just attempted to do the same and found the following:

You can create a Device Channel using code and Windows PowerShell as they are stored in a list
Creating a master page to device channel mapping is not technically available via object model or Windows PowerShell (at least via the APIs that they have provided you)

To create a device channel, you can do this:

using (SPSite site = new SPSite("http://intranet.contoso.com"))
            using (SPWeb web = site.RootWeb)
            {
                SPList list = web.Lists.TryGetList("Device Channels");

                SPListItem li = list.AddItem();
                li["Name"] = "Windows Phone";
                li["ContentType"] = "Device Channel";
                li["Active"] = true;
                //alias can contain no spaces
                li["Alias"] = "WindowsPhone";
                li["Description"] = "The windows phone mobile channel";
                li["Device Inclusion Rules"] = "Windows Phone";
                li.Update();
            }

If you look at how the master page settings page is laid out, it shows you all the device channels and any master pages that are tied to them. When you look at the code, you will find that the settings are converted into a MasterPageMappingsFile object (in the Microsoft.SharePoint.Publishing.Mobile namespace). It inherits from a base class called MappingsFile<T>, both of which are marked as internal and thus you cannot use them. When you review how it builds the list of mappings, it does so using a file called __DeviceChannelMappings.aspx that is stored in the "/_catalogs/masterpage/__DeviceChannelMappings.aspx". It looks like this:

<%@ Reference VirtualPath="~CustomMasterUrlForMapping0" %><%@ Reference VirtualPath="~CustomMasterUrlForMapping1" %><%@ Page Language="C#" Inherits="Microsoft.SharePoint.Publishing.Internal.WebControls.MappingsFileBasePage" %><html xmlns:mso="urn:schemas-microsoft-com:office:office" xmlns:msdt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882"><%@ Register Tagprefix="SharePoint" Namespace="Microsoft.SharePoint.WebControls" Assembly="Microsoft.SharePoint, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
<head>
<meta name="WebPartPageExpansion" content="full" />
<!–[if gte mso 9]>
<SharePoint:CTFieldRefs runat=server Prefix="mso:" FieldList="FileLeafRef"><xml>

<mso:CustomDocumentProperties>
<mso:ContentTypeId msdt:dt="string">0x010100FDA260FD09A244B183A666F2AE2475A6</mso:ContentTypeId>
</mso:CustomDocumentProperties>
</xml></SharePoint:CTFieldRefs><![endif]–>
</head><body><mappings>
<mapping>
    <channelAlias>WindowsPhone</channelAlias>
    <masterUrl href="/_catalogs/masterpage/windowsphone.intranet.master" token="~sitecollection/_catalogs/masterpage/windowsphone.intranet.master" />
</mapping>
<defaultChannelMapping>
    <siteMasterUrl token="~sitecollection/_catalogs/masterpage/seattle.master" href="/_catalogs/masterpage/seattle.master" />
    <systemMasterUrl token="~sitecollection/_catalogs/masterpage/seattle.master" href="/_catalogs/masterpage/seattle.master" />
    <alternateCssUrl token="" href="" />
    <themedCssFolderUrl token="" href="" isthemeshared="false" />
</defaultChannelMapping>
</mappings></body></html>

Now that you know where the values are stored, you can programmatically modify the file using XML tools by downloading the files, changing it and then uploading it. It should be noted that the file format may change in the future and its most likely why they have locked it down from an object model code standpoint.

Enjoy!
Chris

August 20, 2013

Extending the Ceres Engine with custom flows and operators

So what the heck does that title mean? Well, for those of you that are not familiar with Search (which is a majority of you out there). The actual engine is called "Ceres". As in the dwarf planet in our solar system (Wikipedia). Keeping with universe terms, there is also a constellation of nodes in the search engine that make up the universe of bodies in the engine. If you take a minute, you will find several references to Constellation in the various classes inside the assemblies, but enough about the universe, what about extending the Ceres engine?

When it comes to search, many of you are already familiar with the various nodes types that make up the system. This includes:

Admin
Content Processing
Query
Indexing
Analytics

But that's the easy part. and so are the architecture design aspects! This post will take you into a rabbit hole that you may never come out of! For the purpose of this post, we are interested in the Content Processing component AND the Query component. If you dive into the core of the Content Processing component you will find that it is made up of a series of flows. You can find the registered flows in the "C:Program FilesMicrosoft Office Servers15.0SearchResourcesBundles" directory, I will describe what these dlls are and how they get generated later in the post. Here is the full list (in the future I will update this post with what each of these flows purpose is):

Microsoft.ContentAlignmentFlow
Microsoft.CustomDictionaryDeployment
Microsoft.ThesaurusDeployment
Microsoft.CXDDeploymentCaseInSensitive
Microsoft.CXDDeploymentCaseSensitive
Microsoft.PeopleAnalyticsOutputFlow
Microsoft.PeopleAnalyticsFeederFlow
Microsoft.ProductivitySearchFlow
Microsoft.SearchAnalyticsFeederFlow
Microsoft.SearchAnalyticsInputFlow
Microsoft.SearchAnalyticsOutputFlow
Microsoft.SearchAuthorityInputFlow
Microsoft.SearchClicksAnalysisInputFlow
Microsoft.SearchDemotedInputFlow
Microsoft.SearchReportsAnalysisInputFlow
Microsoft.UsageAnalyticsFeederFlow
Microsoft.UsageAnalyticsReportingAPIDumperFlow
Microsoft.UsageAnalyticsUpdateFlow
Microsoft.CrawlerFlow
Microsoft.CrawlerAcronymExtractionSubFlow
Microsoft.CrawlerAlertsDataGenerationSubFlow
Microsoft.CrawlerAliasNormalizationSubFlow
Microsoft.CrawlerComputeFileTypeSubFlow
Microsoft.CrawlerCCAMetadataGenerationSubFlow
Microsoft.CrawlerContentEnrichmentSubFlow
Microsoft.CrawlerDefinitionClassificationSubFlow
Microsoft.CrawlerDocumentSignatureGenerationSubFlow
Microsoft.CrawlerDocumentSummaryGenerationSubFlow
Microsoft.CrawlerHowToClassificationSubFlow
Microsoft.CrawlerLanguageDetectorSubFlow
Microsoft.CrawlerLinkDeleteSubFlow
Microsoft.CrawlerNoIndexSubFlow
Microsoft.CrawlerPhoneNumberNormalizationSubFlow
Microsoft.CrawlerSearchAnalyticsSubFlow
Microsoft.CrawlerTermExtractorSubFlow
Microsoft.CrawlerWordBreakerSubFlow
Microsoft.SharePointSearchProviderFlow
Microsoft.PeopleExpertiseSubFlow
Microsoft.PeopleFuzzyNameMatchingSubFlow
Microsoft.PeopleKeywordParsingSubFlow
Microsoft.PeopleLinguisticsSubFlow
Microsoft.PeopleResultRetrievalAndProcessingSubFlow
Microsoft.PeopleSearchFlow
Microsoft.PeopleSecuritySubFlow
Microsoft.OpenSearchProviderFlow
Microsoft.ExchangeSearchProviderFlow
Microsoft.DocParsingSubFlow
Microsoft.MetadataExtractorSubFlow
Microsoft.AcronymDefinitionProviderFlow
Microsoft.BestBetProviderFlow
Microsoft.QueryClassificationDictionaryCompilationFlow
Microsoft.RemoteSharepointFlow
Microsoft.PersonalFavoritesProviderFlow
Microsoft.QueryRuleConditionMatchingSubFlow
Microsoft.CrawlerDocumentRetrievalSubFlow
Microsoft.CrawlerIndexingSubFlow
Microsoft.CrawlerPropertyMappingSubFlow
Microsoft.CrawlerSecurityInsertSubFlow
Microsoft.OOTBEntityExtractionSubFlow
Microsoft.CustomEntityExtractionSubFlow

The most important flow is the Microsoft.Crawlerflow. This flow is the master flow and defines the order of how all the other flows will be executed. A flow is simply an xml document that defines the flows and operators that should be executed on an item that is processed in the engine. The xml makes up an OperatorGraph. Each Operator has a name and a type attribute. The type attribute is made up of the namespace where the class lives that contains the code for the flow and then the name property of a special attribute added to the class. Each operator is deserialized into an instance of a class as the flow is "parsed". As you review the xml, you should see that the flow of the flow is determined by the "operatorMonkier" that has the name of the next operator that should be executed. The first part of this file looks like the following:

<?xml version="1.0" encoding="utf-8"?>
<OperatorGraph dslVersion="1.0.0.0" name="" xmlns="http://schemas.microsoft.com/ceres/studio/2009/10/flow">
<Operators>

    <Operator name="Init" type="Microsoft.Ceres.ContentEngine.Operators.BuiltIn.Mapper">
      <Targets>
        <Target breakpointEnabled="false">
          <operatorMoniker name="//Operation Router" />
        </Target>
      </Targets>
      <Properties>
        <Property name="expressions" value="{"externalId"="ToInt64(Substring(id, 7))"}"/>
        <Property name="fieldsToRemove" />
        <Property name="adaptableType" value="True" />
      </Properties>
      <OutputSchema>
        <Field name="tenantId" type="Guid" canBeNull="true" expression="IfThenElse(BucketHasField(content, "012357BD-1113-171D-1F25-292BB0B0B0B0:#104"), ToGuidFromObject(GetFieldFromBucket(content, "012357BD-1113-171D-1F25-292BB0B0B0B0:#104")), ToGuid("0C37852B-34D0-418E-91C6-2AC25AF4BE5B"))" />
        <Field name="isdir" type="Boolean" canBeNull="true" expression="NullValue(ToBoolean(GetFieldFromBucket(content, "isdirectory")),false)" />
        <Field name="noindex" type="Boolean" canBeNull="true" expression="NullValue(ToBoolean(GetFieldFromBucket(content, "noindex")),false)" />
        <Field name="oldnoindex" type="Boolean" canBeNull="true" expression="NullValue(ToBoolean(GetFieldFromBucket(content, "oldnoindex")),false)" />
        <Field name="getpath" type="String" expression="IfThenElse(BucketHasField(content, "path_1"), GetStringFromBucket(content, "path_1"), GetStringFromBucket(content, "path"))" />
        <Field name="extrapath" type="String" expression="GetStringFromBucket(content, "path_1")" />
        <Field name="size" type="Int32" expression="TryToInt32(GetFieldFromBucket(content, "size"))" />
        <Field name="docaclms" type="Blob" expression="GetFieldFromBucket(content, "docaclms")" />
        <Field name="docaclsp" type="Blob" expression="GetFieldFromBucket(content, "spacl")" />
        <Field name="docaclmeta" type="String" expression="IfThenElse(BucketHasField(content, "2EDEBA9A-0FA8-4020-8A8B-30C3CDF34CCD:docaclmeta"), GetStringFromBucket(content, "2EDEBA9A-0FA8-4020-8A8B-30C3CDF34CCD:docaclmeta"), GetStringFromBucket(content, "docaclmeta"))" />
        <Field name="docaclgrantaccesstoall" type="Boolean" canBeNull="true" expression="NullValue(ToBoolean(GetFieldFromBucket(content, "grantaccesstoall")),false)" />
        <Field name="externalId" type="Int64" expression=""ToInt64(Substring(id, 7))"" />
        <Field name="sitecollectionid" type="Guid" canBeNull="true" expression="ToGuid(GetStringFromBucket(content, "00130329-0000-0130-C000-000000131346:ows_SiteID"))" />
        <Field name="fallbackLanguage" type="String" expression=""en"" />
        <Field name="Path" type="String" expression="GetStringFromBucket(content, "49691C90-7E17-101A-A91C-08002B2ECDA9:#9")"/>
        <Field name="SiteID" type="String" expression="GetStringFromBucket(content, "00130329-0000-0130-C000-000000131346:ows_SiteID")" />
  <Field name="ContentSourceID" type="Int64" canBeNull="true" expression="IfThenElse(BucketHasField(content, "012357BD-1113-171D-1F25-292BB0B0B0B0:#662"), ToInt64(GetFieldFromBucket(content, "012357BD-1113-171D-1F25-292BB0B0B0B0:#662")), ToInt64(-1))" />
  <Field name="Attachments" type="List<Stream>" canbenull="true" expression="GetFieldFromBucket(content, "attachments")"/>
  <Field name="FileExtension" type="String" expression="IfThenElse(BucketHasField(content, "0B63E343-9CCC-11D0-BCDB-00805FCCCE04:FileExtension"), GetStringFromBucket(content, "0B63E343-9CCC-11D0-BCDB-00805FCCCE04:FileExtension"), "")" />
      </OutputSchema>
    </Operator>

As you can see the first operator that is executed is of the type "Microsoft.Ceres.Evaluation.Operators.Core.Input". This means that if you look in the "Microsoft.Ceres.Evaluation.Operators" namespace, you will find a class that is decorated like the following:

[Serializable, Operator("Input", MinInputCount=0, MaxInputCount=0)]
public class InputOperator : TypedOperatorBase<InputOperator>, IMemoryUsingOperator, IOutputTypeConfigurableOperator
{
…

You should note that the class is marked as serializable and that the Operator attribute has been added with the name "Input". Again, the combination of the namespace of the class and the name of the attribute are used to find the operator when the flow is executed.

One of the flows that I am most interested in is the Microsoft.CrawlerContentEnrichmentSubFlow. As some of you are aware, you can "extend", really don't like that word used in context of Content Enrichment now that I know how to do flow insertion, using a web service to add your own logic to create new crawled properties on items that pass through the engine. You can find more information about content enrichment and examples of using it at http://msdn.microsoft.com/en-us/library/jj163968.aspx. Now, Microsoft is going to tell you that this is the only supported way to extend the Ceres engine. And that is correct. What I am about to show you has never been done outside of Microsoft and if you venture down this path, you do so on your own. Anyway, the problem with the CES is that it is not flexible and it uses stupid old technology called web services. that means it is sending this big ugly xml around on the wire…not JSON. Bummer. That's not the only thing. When you look at the pipeline and all the things you are indexing, if you do not put a trigger on the CES, EVERY single item will be passed to your service. You would then need to have all kinds of logic to determine the type of the item, what properties exist on it by looping through them all and so many other weird bad things it just makes me cringe. Now, if you do put a trigger on it, you are now limiting yourself to implementing a very targeted set of logic. You have no ability to add more than one CES with different triggers with different logic. Huh? Big feature gap here. I'm not a fan. So for people that just don't want to multiple the time it takes to do a crawl by 100 to 1000x over because you implemented CES, you need a better option. A faster option. A more reliable and performant option. One that lives in the engine, not outside of it. If you want to know how to do this…keep reading!

Ok, so this is all simple so far. But how does one add a new flow to the Ceres engine and then implement your own Operators? Well, this is much more difficult than you think!

The first step is to create an operator class that inherits from TypedOperatorBase<T>. Where T is the class name. This is an abstract class and you must implement the method called ValidateAndType. You can see most of this in the operator example above. The next step is to add the Serializable and Operator attributes to the class. Ok, fair enough, now what do we do? If you look at the XML of an operator, you will see that you can implement properties and that those properties are simply deseriabled to the properties in the class. Ok, so add some properties. In my example, I create a class with one property:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Ceres.Evaluation;
using Microsoft.Ceres.Evaluation.Operators;
using Microsoft.Ceres.Evaluation.Operators.PlugIns;

namespace CustomOperator
{
    [Serializable, Operator("CustomOperator")]
    public class CustomOperator : TypedOperatorBase<CustomOperator>
    {
        private string custom = "";

        public CustomOperator()
        {
            this.custom = "Chris Givens was here";
        }

        [Property(Name="custom")]
        public string Custom
        {
            get { return custom; }
            set { custom = value; }
        }

        protected override void ValidateAndType(OperatorStatus status, IList<Microsoft.Ceres.Evaluation.Operators.Graphs.Edge> inputEdges)
        {
            status.SetSingleOutput(base.SingleInput.RecordSetType);
        }
    }
}

Ok, great. So now what do we do? Well, I wasn't sure if the system would just pick up the assembly from the GAC dynamically so I figured, let's just deploy the solution and try to add a flow with the operator in it. Here's how you do that:

Deploy the assembly to the GAC…easy, right-click the project, select "Deploy"

Next, create a new flow (xml file) that uses the operator:

<?xml version="1.0" encoding="utf-8" ?>
<OperatorGraph dslVersion="1.0.0.0" name="CustomFlow" xmlns=" http://schemas.microsoft.com/ceres/studio/2009/10/flow">
<Operators>

Connect to the ceres engine and try to deploy the flow:

Add-PsSnapin Microsoft.SharePoint.Powershell
& "C:Program FilesMicrosoft Office Servers15.0SearchScriptsceresshell.ps1"
Connect-System -Uri (Get-SPEnterpriseSearchServiceApplication).SystemManagerLocations[0] -ServiceIdentity contososp_farm
Connect-Engine -NodeTypes InterActionEngine
$flowname = "CustomFlow"
Remove-Flow $flowname
Get-Content C:CustomOperatorCustomOperator$flowname.xml | Out-String | Add-Flow $flowname
Stop-Flow –FlowName $flowname –ForceAll

You will get the following error that the system cannot find the Operator called CustomOperator.CustomOperator. Bummer. So that didn't work. So how do I "register" my operator with the engine? Well, it turns out that their is so much more that needs to be done than simply creating an operator class. You also need to create several other classes with special attributes attached to them. Sooo…here we go!

First off, you will need to create a Producer class. This producer is really the class that does all the work. The operator is really just a way to get some parameters into the producer. As you can see the Producer inherits from SingleOutputProducer<T>, where T is your operator class. Here is an example of the producer:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Ceres;
using Microsoft.Ceres.Evaluation;
using Microsoft.Ceres.Evaluation.DataModel;
using Microsoft.Ceres.Evaluation.DataModel.Types;
using Microsoft.Ceres.Evaluation.Processing;
using Microsoft.Ceres.Evaluation.Processing.Producers;

namespace CustomOperator
{
    public class CustomProducer : SingleOutputProducer<CustomOperator>
    {
        private CustomOperator op;
        private IRecordSetTypeDescriptor type;
        private IEvaluationContext context;

        public CustomProducer(CustomOperator op, IRecordSetTypeDescriptor type, IEvaluationContext context)
        {
            this.op = op;
            this.type = type;
            this.context = context;
        }

private IUpdateableRecord holder;
//private Item holderItem;

        public override void ProcessRecord(IRecord record)
        {
            this.holder.UpdateFrom(record);

            base.SetNextRecord(record);
        }
    }
}

Next up is to create a NamedPlugInSource. Operators are also called "PlugIns". These plugins must be registered with the system in order for you to use them. If you review all the operator assembiles, you will see that there is always some kind of *PlugInSource class that has the role of adding plugins to the Ceres core system. For my pluginsource, I only have one operator and that is my CustomOperator:

using System;
using System.IO;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Ceres.CoreServices.Services;
using Microsoft.Ceres.CoreServices.Services.Node;

using Microsoft.Ceres.Evaluation.Operators;
using Microsoft.Ceres.Evaluation.Operators.PlugIns;

namespace CustomOperator
{
    [DynamicComponent]
    public class CustomPlugInSource : NamedPlugInSource<OperatorBase>
    {
        public static OperatorBase PlugIn1()
        {
            File.AppendAllText(@"c: empsearch.txt", "PlugIn1");
            return new CustomOperator();
        }

        protected override void AddPlugIns()
        {
            File.AppendAllText(@"c: empsearch.txt", "AddPlugIns");
            Func<OperatorBase> f = PlugIn1;
            base.Add(typeof(CustomOperator),f);
        }
    }
}

Now that you have the plugin built. You will notice that is has been decorated with the "DynamicComponent" attribute. This is where the "Ah-ha" moment kicks in. By adding this attribute to the assembly, Ceres knows that is must start this as a managed component in the system. However, just simply deploying this to the GAC, will not get Ceres to recognize the assembly and load the components. We'll get to that soon, we still have lots more to talk about!

Next up is an Evaluator. An Evaluator is responsible for actually making the call to the producer. In my example I create a class that inherits from ProductEvaluator<T> where T is my CustomOperator. ProductEvaluator is again an abstract class with one method called GetProducer. You must instatiate your producer here and return it. There are many types of producers, but I have not had the time to document all of them as of yet. Soon though!

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Ceres.CoreServices.Services;
using Microsoft.Ceres.CoreServices.Services.Node;
using Microsoft.Ceres.CoreServices.Services.Container;
using Microsoft.Ceres.Evaluation.Processing;
using Microsoft.Ceres.Evaluation.Processing.Producers;
using Microsoft.Ceres.Evaluation.Operators;

using Microsoft.Ceres.Evaluation.DataModel;
using Microsoft.Ceres.Evaluation.Operators.Graphs;

namespace CustomOperator
{
    public class CustomEvaluator : ProducerEvaluator<CustomOperator>
    {
        /*
        protected override IRecordSet SetupOutput(Edge outputEdge, IList<IRecordSet> inputs)
        {
            CustomProducer cp = new CustomProducer();
            return null ;
        }
         */

        protected override IRecordProducer GetProducer(CustomOperator op, Microsoft.Ceres.Evaluation.DataModel.Types.IRecordSetTypeDescriptor type, IEvaluationContext context)
        {
            return new CustomProducer(op, type, context);
        }
    }
}

Next on the list is an EvaluatorBinder. The evaluator binder is responsible for registering an operator with an evaluator. This class will inherit from AbstractEvaluatorBinder and need to implement the AddBoundOperators and BindEvaluator methods:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Ceres.Evaluation.Processing;
using Microsoft.Ceres.Evaluation.Operators;

namespace CustomOperator
{
    public class CustomEvaluatorBinder : AbstractEvaluatorBinder
    {
        protected override void AddBoundOperators()
        {
          base.Add(typeof(CustomOperator));
        }

        public override Evaluator BindEvaluator(OperatorBase op, IEvaluationContext context)
        {
            if (op is CustomOperator)
            {
                return new CustomEvaluator();
            }

            return null;
        }
    }
}

Last on the list is the EvaluatorBinderSource. Similar to a PlugInSource, this will also be decorated with the DynamicComponent attribute which will instantiate and register the evaluators. Here is the binder source:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Ceres.CoreServices.Services;
using Microsoft.Ceres.CoreServices.Services.DependencyInjection;
using Microsoft.Ceres.CoreServices.Services.Node;
using Microsoft.Ceres.CoreServices.Services.Container;
using Microsoft.Ceres.Evaluation.Processing;
using Microsoft.Ceres.Evaluation.Operators;

namespace CustomOperator
{
    [DynamicComponent]
    public class CustomEvaluatorBinderSource : AbstractContainerManaged
    {
        [Exposed]
        public IEvaluatorBinder CieEvaluatorBinder
        {
            get
            {
                this.exampleBinder = new CustomEvaluatorBinder();
                return this.exampleBinder;
            }
        }

private CustomEvaluatorBinder exampleBinder;
}
}

You now have everything you need to add a new flow and operator to the Ceres engine! Kinda. If you deploy the code at this point, you will notice if you try to run the above install script, you will still get the same error! This is because the assemblies only get loaded when you restart the Host Controller service. NOTE: You can read more about the Host controller service in Randy Williams and I's MSPress book on SharePoint due out very soon. Ok, so re-start the service. Try the commands…NO GO…bummer. But I did everything you said Chris! Why doesn't it recognize my operator? Well…going back to my previous statement, Ceres nodes don't look at the entire GAC and analyze every class. That would be WAAAY to expensive. So it only does the one that it is told to do. This was the final magic step that I stumbled upon very luckily.

For each node that is started (via the NodeRunner.exe process), each one is fed its own configuration file that drives the WCF configuration. This file is stored in C:Program FilesMicrosoft Office Servers15.0SearchRuntime1.0
oderunner.exe.config. It is a very generic file, not much going on here. As part of the NodeController code, it will look for another file and feed some special values into the process in addition to the regular app.config file. These files are stored in the Ceres node directory which is in C:Program FilesMicrosoft Office Servers15.0DataOffice ServerApplicationsSearchNodes<RandomNodeID>. Each role that has been assigned to the server will get a directory under this path. Since most of what we are doing is related to the ContentProcessingComponent, let's look there first. If you open and explore this directory, what you will find is a nodeprofile.xml file. It looks like this…tell me if you notice anything interesting:

<?xml version="1.0" encoding="utf-8"?>
<NodeProfile xmlns="http://schemas.microsoft.com/ceres/hostcontroller/2011/08/nodeprofile">
<AutoStart xmlns="">true</AutoStart>
<Stopped xmlns="">false</Stopped>
<Modules xmlns="" />
<Properties xmlns="">
    <Property Key="Managed.Node.Name" Type="string" Value="ContentProcessingComponent1" />
    <Property Key="Managed.SystemManager.ConstellationName" Type="string" Value="A99B1A" />
    <Property Key="Managed.Node.SystemName" Type="string" Value="A99B1A" />
    <Property Key="Managed.SystemManager.ConstellationVersion" Type="int" Value="-1" />
    <Property Key="Managed.Runtime.Version" Type="string" Value="1.0" />
    <Property Key="Managed.Node.LocalSystemManager" Type="bool" Value="False" />
    <Property Key="Managed.Node.ShutdownOnComponentFailed" Type="bool" Value="True" />
    <Property Key="Managed.Node.ProcessPriorityClass" Type="string" Value="BelowNormal" />
    <Property Key="Managed.Node.DynamicAssemblies" Type="string" Value="Microsoft.Ceres.ContentEngine.AnnotationPrimitives, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Bundles, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Component, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.DataModel.RecordSerializer, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Fields, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.LiveEvaluators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.NlpEvaluators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.NlpOperators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Operators.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Parsing.Component, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Parsing.Evaluators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Parsing.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Processing, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Processing.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Properties, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.AliasLookup, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.RecordCache, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.RecordType, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Repository, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Services, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.SubmitterComponent, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Types, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Util, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.ContentEngine.Processing.Mars, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.DataModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.DataModel.Types, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Engine, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Engine.WcfTransport, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators.Core, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Operators.Parsing, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Processing, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Processing.BuiltIn, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.Evaluation.Services, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.DocumentModel, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Admin, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.ContentRouter, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Services, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Utils, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Schema.SchemaCatalogProxy, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Query.MarsLookupComponent, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.FastServerMessages, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchCore.Schema.SchemaCatalog, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.AnnotationStore, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Automata, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Dictionaries, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.DictionaryInterface, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Ese.Interop, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.RichFields, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.RichTypes, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.StringDistance, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.Transformers, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.NlpBase.IndexTokenizer, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.AnalysisEngine.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.SearchAnalytics.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;Microsoft.Ceres.UsageAnalytics.Operators, Version=15.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c;CustomOperator, Version=1.0.0.0, Culture=neutral, PublicKeyToken=7d300eac1b9f50c2" />
    <Property Key="Managed.Node.SearchServiceApplicationName" Type="string" Value="14087e61-67e2-4245-b23d-0e52c6dcf704" />
    <Property Key="Managed.Node.SystemDisplayName" Type="string" Value="0a1ee46f-59f2-49b7-bfca-bb4d20adaf1a" />
    <Property Key="Managed.Node.BasePort" Type="int" Value="17042" />
    <Property Key="Managed.Node.BasePort.4" Type="int" Value="17046" />
    <Property Key="PortShared" Type="bool" Value="True" />
</Properties>
</NodeProfile>

If you guessed the "Managed.Node.DynamicAssemblies" property…then you are very smart! [:D] Yep…that is what we are looking for. Those are the only assemblies that will be loaded into the AppDomain. Only these assemblies will be interrogated for the DynamicComponent attribute. Great! So as you can see, I have added my CustomOperater assembly to the list. Let's try again and run the script. Dang it! NO GO! It still doesn't like my CustomOperator.CustomOperator operator! Grrr….so at this point, I'm really wondering if my assembly is getting laoded…after a browse in the ULS logs…I see these:

08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Inactive] to [Configuring and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiyyo Verbose Microsoft.Ceres.CoreServices.Management.ManagementServer : Registered agent CustomOperator.CustomEvaluatorBinderSource.ComponentManager of type Microsoft.Ceres.CoreServices.Services.Container.IComponentManagerManagementAgent
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Configuring] to [Configured and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Configured] to [Resolving and eventSent=True]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Resolving] to [Readying and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Readying] to [Ready and eventSent=True]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Ready] to [Activating and eventSent=False]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomPlugInSource) : CustomPlugInSource moved from [Activating] to [Active and eventSent=True]
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiywj Medium ComponentManager(CustomOperator.CustomPlugInSource) : CustomOperator.CustomPlugInSource [Active] started
08/20/2013 22:32:32.18 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x0784 Search                        Search Platform Services      aiyv9 Verbose ComponentManager(CustomOperator.CustomPlugInSource) : ***** QUEUESENTINEL finished task for CustomOperator.CustomPlugInSource: CustomOperator.CustomPlugInSource[Active]state Active
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Inactive] to [Configuring and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Configuring] to [Configured and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Configured] to [Resolving and eventSent=True]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Resolving] to [Readying and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Readying] to [Ready and eventSent=True]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Ready] to [Activating and eventSent=False]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywq High    ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomEvaluatorBinderSource moved from [Activating] to [Active and eventSent=True]
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiywj Medium ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : CustomOperator.CustomEvaluatorBinderSource [Active] started
08/20/2013 22:32:32.20 NodeRunnerContent1-0a1ee46f-59f (0x3994) 0x39E4 Search                        Search Platform Services      aiyv9 Verbose ComponentManager(CustomOperator.CustomEvaluatorBinderSource) : ***** QUEUESENTINEL finished task for CustomOperator.CustomEvaluatorBinderSource: CustomOperator.CustomEvaluatorBinderSource[Active]state Active

Ok…they ARE being loaded. So what the hell is going on? Well…the clue WAS in the logs files. After running the PowerShell to attempt to add the flow, I noticed something. It was the name of the component that is actually being used to register a flow. Its called QueryProcessingComponent1. Well ok, so it seems that even though the content processing node does all the work, the query processing component manages all the registration of the plugins and operators. After going back to the node directory, I find the QueryProcessingComponent1 directory and find that it too has a NodeProfile.xml file. Bingo. Adding the assembly to the property and restarting the host controller one more time, I again attempt to add a custom flow, with a custom operator.

YYYYYEEEESSSS!!!! NO ERROR…………..I successfully inserted my flow and operator into the Ceres engine! Now, what part is missing? Well, even though the flow is now installed and working, it is not a part of the main flow ("Microsoft.CrawlerFlow"). I would need to insert the flow into that main file and then redeploy it. The main issue with that, is that not all of the operators are recognized by the system. Yeah, weird I know. This is part of the installationdeployment of the search service application and is there by default. if you ever want to make changes, you would need to add all the possible assemblies to the query processing component and then update the main flow.

In terms of debugging, you can attach to the NodeRunner.exe processes and debug your operator and evaluators. Easy.

Now for some clean up. All those bundles of flows at the top of this post. How did they get there? Well, what happens is each time you upload a flow, it will generate a new assembly with the flow added to it as a resource. If you were to reflect on any of the assemblies above, you can get the flow xml out of the assembly. But this is also easily done using the Windows PowerShell commands above.

I will be posted all the code for this project on code.msdn.microsoft.com. You can use it as a starting point for implementing your own flows and operators. But you are probably asking, why would I do something that is not supported. Well, its the same reason you want to keep your job. The customer wants high performance and needs to implement way more than the Content Enrichment Service can provide and saying no will stop any chance you have of completing an incredibly cool and awesome project. Now, why is this not supported if you CAN do it? Well, as you can see, it is VERY complex. Only a few people in the world are going to be able to build these, deploy them and successfully use them. So you are still asking yourself…why did you post this if we can't really do it…great question!

BECAUSE I WANT IT SUPPORTED. If we band together and find various use cases for doing this, the product team will have no choice but to train the Level I, II, and III support people on how to troubleshoot these. As of right now, it is simply unsupported from the fact that the support people don't even know what a flow and operator is when it comes to supporting SharePoint Search (update: met with the search team and the COE support team *is* familiar with flows so were one step closer to support). It would be my goal to get some ISVs to start playing around with creating custom flows and operators to make SharePoint Search a BEAST (not that is already isn't cuz its the best on the market right now, sorry Google Appliance but you suck big time)! So…there you have it. Do with it what you will, have fun, be smart and as always…enjoy!

Chris

August 10, 2013

BCS, OData and Subscriptions – How to get it working!

So what have I been working on for the past two weeks? Well, other than consulting clients, books and working on my garden, I have also been involved with the Microsoft Learning SharePoint 2013 Advanced Development Microsoft Official Curriculum (MOC) course 20489 that will be available later in the year (sorry no link just yet but it will be here when it is released). I was able to finish up two chapters on Search quickly as that is one of my main fortes, but then decided to take what I though was the middle of two Business Connectivity Services (BCS) chapters. For those of you not familiar with BCS, you can find a great overview here. It turns out, the module was the hardest one! Why? Because it covers things that no one has ever done before (outside of the product team that is).

So what is the big deal about what I worked on? You are probably saying to yourself…BCS has been around for a while right? Well, yes this is very true, and there are several great posts about how to expose external data using SharePoint Designer and Visual Studio using the various BDC model types (Database, WCF, .NET Connectivity and Custom). You can also find how to implement stereotyped methods that support CRUD methods and search indexing (link here). Given all that content, there were a game changing set of features that were added to BCS in SharePoint 2013 that add a whole new level of complexity. These features include:

There are plenty of posts on OData in general (this one from MSDN is pretty awesome if you are just getting started) and a few posts on how to setup a BDC OData model. And although my fellow SharePoint MVP Scot Hillier did a presentation on the subscriber model at the last SharePoint Conference it was only in context of a database model. When it comes to integrating the two features (OData and the subscriber methods) together, that is where a massive black hole exists and is the focus of this blog post.

The first step to getting this whole thing to work is to create an OData service. This is very simple with the tools provided by Visual Studio and steps to do this are provided in this MSDN post

The next step is to build your basic BCS model using the new item template wizard provided in Visual Studio 2012. This has also been nicely blogged about by several of my colleagues and does have an article on MSDN. The important thing to note about the MSDN article I reference is that it is using an OData feed that is hosted by http://services.odata.org. Since you do not own this service, you will not be able to extend it to implement the subscribe and unsubscribe methods that I discuss later in this post. Therefore, you can follow the steps in the article, but use a local instance of your OData service.

Once the service has been generated, you must add some supporting methods to your OData service to accept data from SharePoint when a subscription occurs. There are some gotchas to this. Currently there is no real guidance on how to set this up properly. The little that does exist will point you to mixed signals as to how to successfully setup the communication layers. In my example below, you will see that I am using a GET for the http method. This was the only successful way that I was able to get the method parameters to populate in the web method in the OData service. As you will see later, there are also some very important BDC method properties that must be set in order for all of this to work:

[WebGet]
        public string Subscribe(string DeliveryURL, int EventType, string EntityName, string SelectColumns)
        {
            //HttpRequest req = System.Web.HttpContext.Current.Request;

// Generate a new Guid that will function as the subscriptionId.
string subscriptionId = Guid.NewGuid().ToString();

if (DeliveryURL == null || EventType == null || EntityName == null || SelectColumns == null)
throw new Exception(""Missing parameters");

// This sproc will be used to create the subscription in the database.
string subscribeSproc = "SubscribeEntity";

string sqlConn = "Data Source=.;Initial Catalog=Northwind;uid=sa;pwd=Pa$$w0rd";

            // Create connection to database.
            using (SqlConnection conn = new SqlConnection(sqlConn))
            {
                SqlCommand cmd = conn.CreateCommand();
                cmd.CommandText = subscribeSproc;
                cmd.CommandType = CommandType.StoredProcedure;

                cmd.Parameters.Add(new SqlParameter("@SubscriptionId", subscriptionId));
                cmd.Parameters.Add(new SqlParameter("@EntityName", EntityName));
                cmd.Parameters.Add(new SqlParameter("@EventType", EventType));
                cmd.Parameters.Add(new SqlParameter("@DeliveryAddress", DeliveryURL));
                cmd.Parameters.Add(new SqlParameter("@SelectColumns", SelectColumns));

                try
                {
                    conn.Open();
                    cmd.ExecuteNonQuery();
                }
                catch (Exception e)
                {
                    throw e;
                }
                finally
                {
                    conn.Close();
                }

                return subscriptionId;
            }
        }

[WebGet]
        public void Unsubscribe(string subscriptionId)
        {
            HttpRequest req = System.Web.HttpContext.Current.Request;

            // This sproc will be used to create the subscription in the database.
            string subscribeSproc = "UnsubscribeEntity";

string sqlConn = "Data Source=.;Initial Catalog=Northwind;uid=sa;pwd=Pa$$w0rd";

cmd.Parameters.Add(new SqlParameter("@SubscriptionId", subscriptionId));

                try
                {
                    conn.Open();
                    cmd.ExecuteNonQuery();
                }
                catch (Exception e)
                {
                    throw e;
                }
                finally
                {
                    conn.Close();
                }
            }
        }

On the BCS side, you need to add the stereotyped methods that will send the data to the web methods you just created in the last step. This includes the EntitySubscriber and EntityUnsubscriber methods. First the let's review the EntitySubscriber method. In the table below, you will notice that I am sending the OData web method parameters in the querystring. You can use the '@' parameter notation just like in regular BDC Models to token replace the values. You should also use HTML encoded '&' to signify the ampersand (this was one of the things that took me a while to figure out). Notice the various method parameters. They include:

ODataEntityUrl – this is appended to the ODataServiceURL property of the LobSystemInstance (note that later when doing an explicit subscription call, the notification callback url will NOT be used)
ODataHttpMethod – the type of HTTP method you will perform (GET, POST, MERGE, etc). I was never able to get POST to work with a Visual Studio generated OData layer, more on that later.
ODataPayloadKind – This is one of the more confusing aspects of OData. You can find the enumeration for the ODataPayloadKind here, but there is very little documentation on how it works between SharePoint and the custom methods you generate on the OData service side. It took me forever to figure out that the "Entity" payload just doesn't work. After running through just about every permutation of Http methods, payloads and formats, I finally found a working combination with the "Property" payload
ODataFormat – This was another painful setting to determine. When you create your OData service, it is expecting a very specific Content-Type http header to be sent, this header is based on the version of Visual Studio you have. I learned this the hard way, but things started to make sense for me after I reviewed this awesome post about how the OData service generation and layers works in Microsoft world and how to customize its behavior after generating it. For more information on OData supported version, check out this post. In several examples, you may see that the format is set to "application/atom+xml". Well, that format is not supported in the OData service! What you will end up with is an http exception being sent to the calling client (in this case SharePoint) that says "Unsupported media type". This is very unfortunate. Why? Because the error occurs last in the call stack of the web method…AFTER your web method code has run and created the subscription successfully! In order to catch this type of event, you must override the HandleException method of the OData service and rollback any subscriptions that were created by using some kind of instance variable! This would apply to anything that happens that would result in an error as the response is being sent back to the client.
ODataServiceOperation – still haven't figured out what this does!
NotificationParserType – this will be explored more below

Here is the working method XML for the Subscribe method:

<Method Name="SubscribeCustomer" DefaultDisplayName="Customer Subscribe" IsStatic="true">
              <Properties>
                <Property Name="ODataEntityUrl" Type="System.String">/Subscribe?DeliveryURL='@DeliveryURL'&EventType=@EventType&EntityName='@EntityName'&SelectColumns='@SelectColumns'</Property>
                <Property Name="ODataHttpMethod" Type="System.String">GET</Property>
                <Property Name="ODataPayloadKind" Type="System.String">Property</Property>
                <Property Name="ODataFormat" Type="System.String">application/json;odata=verbose</Property>
                <Property Name="ODataServiceOperation" Type="System.Boolean">false</Property>
              </Properties>
              <AccessControlList>
                <AccessControlEntry Principal="NT AuthorityAuthenticated Users">
                  <Right BdcRight="Edit" />
                  <Right BdcRight="Execute" />
                  <Right BdcRight="SetPermissions" />
                  <Right BdcRight="SelectableInClients" />
                </AccessControlEntry>
              </AccessControlList>
              <Parameters>
                <Parameter Direction="In" Name="@DeliveryURL">
                  <TypeDescriptor TypeName="System.String" Name="DeliveryURL" >
                    <Properties>
                      <Property Name="IsDeliveryAddress" Type="System.Boolean">true</Property>
                    </Properties>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="In" Name="@EventType">
                  <TypeDescriptor TypeName="System.Int32" Name="EventType" >
                    <Properties>
                      <Property Name="IsEventType" Type="System.Boolean">true</Property>
                    </Properties>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="In" Name="@EntityName">
                  <TypeDescriptor TypeName="System.String" Name="EntityName" >
                    <DefaultValues>
                      <DefaultValue MethodInstanceName="SubscribeCustomer" Type="System.String">Customers</DefaultValue>
                    </DefaultValues>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="In" Name="@SelectColumns">
                  <TypeDescriptor TypeName="System.String" Name="SelectColumns" >
                    <DefaultValues>
                      <DefaultValue MethodInstanceName="SubscribeCustomer" Type="System.String">*</DefaultValue>
                    </DefaultValues>
                  </TypeDescriptor>
                </Parameter>
                <Parameter Direction="Return" Name="SubscribeReturn">
                  <TypeDescriptor Name="SubscriptionId" TypeName="System.String" >
                    <Properties>
                      <Property Name="SubscriptionIdName" Type="System.String">SubscriptionId</Property>
                    </Properties>
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Type="EventSubscriber" ReturnParameterName="SubscribeReturn" ReturnTypeDescriptorPath="SubscriptionId" Default="true" Name="SubscribeCustomer" DefaultDisplayName="Customer Subscribe">
                  <AccessControlList>
                    <AccessControlEntry Principal="NT AuthorityAuthenticated Users">
                      <Right BdcRight="Edit" />
                      <Right BdcRight="Execute" />
                      <Right BdcRight="SetPermissions" />
                      <Right BdcRight="SelectableInClients" />
                    </AccessControlEntry>
                  </AccessControlList>
                </MethodInstance>
              </MethodInstances>
            </Method>

Next is the unsubscribe method, notice how SharePoint must pass back the subscription id that lives in the external system. The name of the SubscriptionIdName property will always be SubscriptionId. This subscription id must be saved somewhere, but the question is…where?:

<Method Name="UnSubscribeCustomer" DefaultDisplayName="Customer Unsubscribe">
              <Properties>
                <Property Name="ODataEntityUrl" Type="System.String">/UnSubscribe?SubscriptionId='@SubscriptionId'</Property>
                <Property Name="ODataHttpMethod" Type="System.String">GET</Property>
                <Property Name="ODataPayloadKind" Type="System.String">Property</Property>
                <Property Name="ODataServiceOperation" Type="System.Boolean">false</Property>
              </Properties>
              <AccessControlList>
                <AccessControlEntry Principal="NT AuthorityAuthenticated Users">
                  <Right BdcRight="Edit" />
                  <Right BdcRight="Execute" />
                  <Right BdcRight="SetPermissions" />
                  <Right BdcRight="SelectableInClients" />
                </AccessControlEntry>
                <AccessControlEntry Principal="Contosodomain users">
                  <Right BdcRight="Edit" />
                  <Right BdcRight="Execute" />
                  <Right BdcRight="SetPermissions" />
                  <Right BdcRight="SelectableInClients" />
                </AccessControlEntry>
              </AccessControlList>
              <Parameters>
                <Parameter Name="@SubscriptionId" Direction="In">
                  <TypeDescriptor Name="SubscriptionId" TypeName="System.String">
                    <Properties>
                      <Property Name="SubscriptionIdName" Type="System.String">SubscriptionId</Property>
                    </Properties>
                  </TypeDescriptor>
                </Parameter>
              </Parameters>
              <MethodInstances>
                <MethodInstance Name="UnSubscribeCustomer" DefaultDisplayName="Customer
             Unsubscribe" Type="EventUnsubscriber" Default="true">
                  <AccessControlList>
                    <AccessControlEntry Principal="NT AuthorityAuthenticated Users">
                      <Right BdcRight="Edit" />
                      <Right BdcRight="Execute" />
                      <Right BdcRight="SetPermissions" />
                      <Right BdcRight="SelectableInClients" />
                    </AccessControlEntry>
                  </AccessControlList>
                </MethodInstance>
              </MethodInstances>
            </Method>

Now that those items are setup, you need to deploy your BCS model and set permissions. This is very common activity so I'll skip the details in this blog post, however I will say that it is annoying that the user that uploads the model is not automatically added (or have an option somewhere to add them on the import page) as a admin with permissions to the model and methods [:(]

Now that the model is deployed, the next step is to enable a feature that enables subscription support, which brings us back to the question brought up before…where does SharePoint store the subscription id of the external system? A list of course! To create this list, there are two features of which you can enable. One is called BCSEvents, the other is called ExternalSubscription. The funny thing about these two features and their relationship is that the BCSEvents feature is made up of a feature activation receiver. That receiver has only one goal: To activate the ExternalSubscription feature. In addition to this interesting design, you will find that the BCSEvents is a hidden feature whereas the ExternalSubscription feature is actually visible in the web features settings page. What does the ExternalSubscription feature do? It creates our list of course! This list is called "External Subscriptions Store". This is a hidden list and can be unhidden using PowerShell, but it exists in the "_private/ExtSubs" folder and has no views from which you can view the data, so again Windows PowerShell is the way to go if you want to see what lives in the list. Here is a screen shot of the columns of the list:

Next you need to create a subscription. This can be done explicitly or implicitly. The explicit way is to make a call to the entity's subscribe method as shown here (as previously pointed out above, the notification callback url is ignored in an OData Model):

function SubscribeEntity() {
    var notificationCallback = new SP.BusinessData.Runtime.NotificationCallback(context, "http://localhost:19739/northwind.svc");
    var url = web.get_url();
    notificationCallback.set_notificationContext(url);
    context.load(notificationCallback);
    var subscription = entity.subscribe(1, notificationCallback, "administrator@contoso.com", "SubscribeCustomer", lobSystemInstance);
    context.load(subscription);
    context.executeQueryAsync(OnSubscribeSuccess, failmethod);
}

//these are the helper methods and variables

var context;
var web;
var user;
var entity;
var lob;
var lobSystemInstance;
var lobSystemInstances;

// This code runs when the DOM is ready and creates a context object which is needed to use the SharePoint object model
$(document).ready(function () {
    context = SP.ClientContext.get_current();
    web = context.get_web();
    context.load(web);

entity = web.getAppBdcCatalog().getEntity("NorthwindModel", "Customers");
context.load(entity);

lob = entity.getLobSystem();
context.load(lob);

lobSystemInstances = lob.getLobSystemInstances();
context.load(lobSystemInstances);

context.executeQueryAsync(GetLobSubscribesystemInstance, failmethod);
});

// Initialize the LobSystemInstance.
function GetLobSubscribesystemInstance() {
    var $$enum_1_0 = lobSystemInstances.getEnumerator();
    while ($$enum_1_0.moveNext()) {
        var instance = $$enum_1_0.get_current();
        lobSystemInstance = instance;
        context.load(lobSystemInstance);
        break;
    }
    context.executeQueryAsync(SubscribeEntity, failmethod);
}

Subscriptions can be one of three types (you can learn more about event types here):

ItemAdded (1)
ItemUpdated (2)
ItemDeleted (3)

Note that there is no event type that supports ItemAdding, ItemUpdating or ItemDeleting. This means you cannot cancel the insertion, update or deletion in the external source, you can only expect to receive notification after the event has occurred.

The implicit way is to create an alert or to setup an event receiver. This means you should setup an external list pointing to your OData model. You can then use the ribbon to create an alert which will in turn execute the web method call to create the subscription. Note that you must setup your outgoing email settings on your farm, or the alert ribbon button will not display! If an error occurs when creating an event receiver, you will be passed the web method exception from your OData service. This can be very helpful for troubleshooting.

NOTE: When you create an external list, there are several items that seems to get cached in the external list's properties that will require you to delete the list and then re-create it. This means that as you are testing your solution, you should create a Windows PowerShell script that will remove your BDC model, re-deploy it, remove the external list and then add it back.

Once this has all been completed, you can now start telling SharePoint that things have changed. As much work as we have done to this point, it is really rather simple compared to the amount of work needed for this component of the eco-system. There are several approaches you could take to do this:

Triggers on the database to populate a table monitored by SQL Server to send events directly to SharePoint
Triggers on the database to populate a table monitored by a windows service
No triggers and just a simple row timestamp monitoring that checks for any insertsupdatesdeletes and sends the notification
Code that sends changes to an event queue like MSMQ or BizTalk that will then send it to SharePoint

Each of these have advantages and drawbacks in terms of time and complexity. No matter what, you need some component that will tell SharePoint that something has changed. In the code samples I provide, you have a simple console application that will allow you to send the notification to SharePoint for testing purposes.

So now that you have something that can send a message to SharePoint, what does that message look like? This process of communication is un-documented anywhere, until now, and is the real meat of this post! It turns out that there are two message parsers that come out of the box with SharePoint. These include an IdentityParser and an ODataEntryContentNotificationParser. The difference between the two is that one only tells SharePoint that a set of identities has changed and the other actually can pass the changed properties of the item to SharePoint. Both requires a completely different style of ATOM message to be sent.

In the case of the IdentityParser, it is looking for a message that looks like the code snippet below. This particular piece of XML must have a valid XPath to "/a:feed/a:entry/a:content/m:properties/b:BcsItemIdentity". If it does not, then any call to "retrieve the item" in your event receiver will fail. The message will be received and the event receiver will execute as long as you don't make calls to the various properties that will not be available without the id. You should also be aware that none of the other items that live outside of the XPath are ever looked at and can be anything you like as they are not validated or used:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<feed xml:base="http://services.odata.org/OData/OData.svc/"
xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices"
xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata"
xmlns:b="http://schemas.microsoft.com/bcs/2012/"
xmlns="http://www.w3.org/2005/Atom">
<entry>
<title type="text">Customers</title>
<id>http://www.northwind.com/customers</id>
<author>
<name>External System</name>
</author>
<content type="application/xml">
<m:properties>
<b:BcsItemIdentity m:type="Edm.String"><CustomerID>ALFKI</CustomerID></b:BcsItemIdentity>
<d:Name>Customer</d:Name>
</m:properties>
</content>
</entry>
</feed>

In the case of the ODataEntryContentNotificationParser, you must pass an XML message that has a valid XPath to "/a:entry/a:link/m:inline/a:entry". The XML node in this XPath must itself be a valid ATOM message. Again, everything that is outside the XPath seems to be ignored and only the embedded ATOM Message is used:

<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<atom:entry xml:base="http://sphvm-92723:90/WcfDataService2.svc" xmlns="http://www.w3.org/2005/Atom" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns:atom="http://www.w3.org/2005/Atom">
<atom:category term="NorthwindModel.EntitySubscribe" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme"/>
<content type="application/xml">
     <m:properties>
      <d:SubscriptionId m:type="Edm.Int32">1</d:SubscriptionId>
      <d:EntityName>Customers</d:EntityName>
      <d:DeliveryURL>{11}</d:DeliveryURL>
      <d:EventType m:type="Edm.Int32">{12}</d:EventType>
      <d:UserId m:null="true" />
      <d:SubscribeTime m:type="Edm.Binary">AAAAAAAABE4=</d:SubscribeTime>
      <d:SelectColumns>*</d:SelectColumns>
    </m:properties>
</content>
<id>OuterId</id>
<atom:id>http://sphvm-92723:90/WcfDataService2.svc/EntitySubscribes(1)</atom:id>
<atom:link href="EntitySubscribe(1)" rel="self" title="EntitySubscribe"/>
<atom:link href="Customers(2147483647)" rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/customers" type="application/atom+xml;type=entry">
    <m:inline>
      <entry xml:base="http://sphvm-92723:90/WcfDataService2.svc/" xmlns:d="http://schemas.microsoft.com/ado/2007/08/dataservices" xmlns:m="http://schemas.microsoft.com/ado/2007/08/dataservices/metadata" xmlns="http://www.w3.org/2005/Atom">
        <id>http://sphvm-92723:90/WcfDataService2.svc/Customers('57849')</id>
        <title type="text" />
        <updated>2012-04-30T11:50:20Z</updated>
        <author>
        <name />
        </author>
        <link rel="edit" title="Customer" href="Customers('57849')" />
        <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/Orders" type="application/atom+xml;type=feed" title="Orders" href="Customers('57849')/Orders" />
        <link rel="http://schemas.microsoft.com/ado/2007/08/dataservices/related/CustomerDemographics" type="application/atom+xml;type=feed" title="CustomerDemographics" href="Customers('57849')/CustomerDemographics" />
        <category term="NorthwindModel.Customer" scheme="http://schemas.microsoft.com/ado/2007/08/dataservices/scheme" />
        <content type="application/xml">
          <m:properties>
            <d:CustomerID>{0}</d:CustomerID>
            <d:CompanyName>{1}</d:CompanyName>
            <d:ContactName>{2}</d:ContactName>
            <d:ContactTitle>{3}</d:ContactTitle>
            <d:Address>{4}</d:Address>
            <d:City>{5}</d:City>
            <d:Region>{6}</d:Region>
            <d:PostalCode>{7}</d:PostalCode>
            <d:Country>{8}</d:Country>
            <d:Phone>{9}</d:Phone>
            <d:Fax>{10}</d:Fax>
          </m:properties>
        </content>
      </entry>
    </m:inline>
</atom:link>
<title>New Customer entry is added</title>
<updated>2011-07-12T09:21:53Z</updated>
</atom:entry>

In addition to the two out of the box parsers, there is a setting that specifies "Custom". By implementing our own NotificationParser, we can format the message in a much more simple and efficient way such as JSON. The main method to implement is the ChangedEntityInstance method. As part of this parser, you will be passed the message byte array in the initialization and it would be your responsibility to parse the message and pass back the entity instance.

public abstract class NotificationParser
{
    // Methods
    protected NotificationParser()
    {
    }

    public void Initialize(NameValueCollection headers, byte[] message, IEntity entity, ILobSystemInstance lobSystemInstance)
    {
        if (message == null)
        {
            message = new byte[0];
        }
        if (entity == null)
        {
            throw new ArgumentNullException("entity");
        }
        if (lobSystemInstance == null)
        {
            throw new ArgumentNullException("lobSystemInstance");
        }
        this.NotificationHeaders = headers;
        this.NotificationMessage = message;
        this.Entity = entity;
        this.LobSystemInstance = lobSystemInstance;
    }

    // Properties
    public virtual IEntityInstance ChangedEntityInstance
    {
        get
        {
            Identity changedItemIdentity = this.ChangedItemIdentity;
            return this.Entity.FindSpecific(changedItemIdentity, this.LobSystemInstance);
        }
    }

    public abstract Identity ChangedItemIdentity { get; }
    protected IEntity Entity { get; private set; }
    protected ILobSystemInstance LobSystemInstance { get; private set; }
    public NameValueCollection NotificationHeaders { get; private set; }
    public byte[] NotificationMessage { get; private set; }
}

Summary:

Now that you have all the pieces, you can download the code I have placed on the code.msdn.microsoft.com site here. This code has a BCS OData model fully working with the subscriber methods. As code generation techniques have become more common place, OData layers generated via Visual Studio are more common as well. It will be well worth implementing these new BDC method stereotypes in your OData model and in your OData services to provide the ability to be notified when data changes in your remote systems!

July 8, 2013

SharePoint’s Navigation Struggles – A Bit of History

SharePoint has always struggled with navigation. Every customer I have been to we have either torn out the navigation completely, or had to customize it in some way to achieve their goals. SharePoint 2013 is no exception to these navigation woes, and if anything, we have taken a step backwards. In looking at what customer's want, I have found there are three types of navigation:

Global-global – this navigation exists across the top of all sites in a SharePoint farm and is exactly the same.
Local-global – this navigation is for the site collection navigation and is the same across the site collection
Local-local – this navigation is for the site navigation and shows important lists and pages on a site

To be clear, SharePoint has never had "Global-Global" navigation. It has however had the "Local-global" and "local-local" navigation, with some helper tools for inter-site collection navigation. In order to fully point out this backwards momentum, let's take a look at some of the previous out of the box UIs:

2007:

As you can see we started with some nice tabbed navigation, with the quick launch. It was nice and simple, we also had the breadcrumb to get us back up to anywhere in the site tree. However, to really get valuable navigation, you had to enable the
PublishingSite and PublishingWeb features to get the fly out navigation
of subsites:

2010:

In 2010, we lost the breadcrumb control that was directly above the place holder main area, but we gained the folder icon in above the ribbon. Again, we have to enable the publishing features to get any real value out of the local-global.

2013:

In 2013, the local-global was moved from directly below the ribbon, to inside the ribbon. We also lost the folder icon.

As you can see, we have lost (removed from the master pages yet still exist in the code base) some great functionality over the years.

It had seemed that Metadata navigation was in many of our minds (based on marketing hype) the solution for the "global-global" navigation problem that SharePoint has had for many years, but alas it is not:

Unfortunately, if you have actually tried to implement it, you will run into these sets of errors:

Trying to use the term store more than once:

Using Windows PowerShell to set it:

$navSettings.CurrentNavigation.Source = 1;
$navSettings.CurrentNavigation.TermStoreid = new-object System.Guid("6ffccd26-5aba-44a5-83a9-60a468261054");
$navSettings.CurrentNavigation.TermSetId = new-object System.Guid("420d7ef6-6040-4138-8e15-1d04773955ba")
$navSettings.GlobalNavigation.Source = 1;
$navSettings.GlobalNavigation.TermStoreid = new-object System.Guid("6ffccd26-5aba-44a5-83a9-60a468261054");
$navSettings.GlobalNavigation.TermSetId = new-object System.Guid("420d7ef6-6040-4138-8e15-1d04773955ba")
$navSettings.Update()

This gets you the dreaded "Error loading navigation: The Managed Navigation term set is improperly attached to the site":

So where does that leave us? It means we have to move the Local-Global of 2013 from the ribbon, back to where it was in 2010 (directly below the ribbon) and then implement our own global navigation provider in the ribbon, we also add back the breadcrumb to the content placeholder main:

How did I do this? Well, its not pretty. Especially when you realize you have to make your own custom master page, not only that, but just about every site definition has some variation on the basic seattle.master. This shows up in the Search Center and My Site templates. Which means you pretty much have to implement a custom master page for each one (which I have done in 2013). Although it is a lot of work, it is well worth it when your customer gives you the thumbs up and is actually able to navigate the sites inside and outside of SharePoint with ease! A few steps to get here:

Create a custom navigation provider that points to a global list with a hierarchy of elements (or where ever)
Remove the ribbon navigation (local-global), replace with your navigation provider menu control
Put the removed ribbon navigation directly below the ribbon, add the older CSS to give it the top and bottom border
Add the breadcrumb directly above the content place holder main area

In a second spin…what does that mean for O365 customers that want to migrate their intranet or anything else they did to SharePoint Online? Its not good I'm afraid. You aren't allowed to push your own code, so the navigation providers are out. That means you won't get global navigation in SharePoint online unless you figure out a way to do it via a custom <DIV> that is populated based on Javascript methods from some data source (preferably from a SPList via REST). But now you have to implement all the flyout code (unless their is some reliable and simple SDK you can find on the internet to do it and plug into your SP Rest calls and hope that SharePoint CSS doesn't mess with it). And don't forget about the lovely security in JavaScript that prevents cross domain calls.

One other option I thought about for the global-global was to put it in the first row of the 2013 page. I had lots of issues with the div tags and flyouts of the basic asp and SharePoint menu's. Although I'm sure I could figure it out at some point, it ended up being much easier just to put the global-global in the ribbon and move the local-global down.

So, with this little bit of education, hopefully we can get someone in the product team to get serious about providing this type of functionality to meet our customer's needs (global-global) without us having to do anything. I still think MMS is the way to go, but I'd guess their are issues with doing this in O365 and that's why it was limited to one site collection.

Chris

July 8, 2013

Taking Office Web Apps 2013 and SharePoint 2013 integration one step further

Office Web Apps and SharePoint are integrated in several very cool ways. For example, you can see the callout menu on a document library to create new documents:

You can view document previews in the callout of a document:

You can see the document preview in the callout of a search result:

One of the things I noticed right off the bat with Office Web Apps 2013 was that the call out menu has the same options for all office content types of a certain type. This causes some issues when users try to click on the links.

An example of this is the "Follow" link. If you click on the "Follow" link of a document that is stored in a file share, it will error with the following:

I'm sure you could get into the display template and remove that action, but it wasn't as important as some of the other items on my plate when it comes to OWA and SharePoint Search integration. For example, you will also notice that you do not get the ability to view a document that lives in the file share in the thumbnail previewer. I find this unacceptable and so did my customer! So…I started out on the path to figure out how to get it working! Here's what I came up with.

The first thing I thought was cool about Office Web Apps is the ability to "embed" documents in your web pages that live pretty much anywhere. If you open the "http://svr-owa/op/generate.aspx" page, you will see you have the ability to create an embedded link:

Once you create the link to a fileshare document you can put it on any of your HTMLSharePoint pages. However, after creating the link, if you try to open it, you will typically get a "File not found" error:

Turns out this is the main wrapper error around just about anything error that happens in OWA. The main reason that the files won't display is because of permission issues on the Office Web App server. You see, it doesn't open the file as you, it opens it as the Office Web Apps service identity! You can find out what your identity is by opening the IIS Manager on the OWA server and looking for the OpenFromUrlWeb app pool:

By default, this account is set to "Network Service". This account can't do much with secured network based resources unless you assign it those permissions. This means doing the whole give DOMAINCOMPUTERNAME$ access to your file share and all the OWA servers in your OWA farm. This then implies that any software that is running on your OWA servers will then have access to your file share. If only OWA is installed, then you should be ok, but don't forget about all the other service that are running as NETWORK SERVICE on the services applet. I'm not a big fan of this and in my case, I made it a specifc OWA domain account that has at least read access to the shares that contain your data. NOTE: This change is not supported by Microsoft, but let's be clear about what unsupported means. There are two types of unsupported features, ones they don't have the scripts for at the first level of support, and the ones they do have at second and third levels of support. In my eyes, this falls into the first category and is a simple change but they have not trained anyone on how to do this or troubleshoot it, so this is at your own discretion. So why does OWA work like this? The reason lies in the way that Office Web Apps must access a file
in order to render it. When it is stored in SharePoint, SharePoint
will pass an OAuth access token that OWA can use to access the file as the
user. This will always ensure that OWA accesses a file that the user can access. When accessing through the OpenFromUrl means, it has to access it
as itself using regular windows auth. This has some security implications. Search does only show files the user has access too, so that isn't really a security hole when using the method below, but where things do get interesting if a savvy user figures out how to construct the web page viewer url to a file they don't have access too. This can elevate their privileges and allow them to look at a file but not change it. In this case, you should place a "Deny" ACL on the OWA account (whether domain or network service) to prevent it from reading the document.

If you aren't comfortable in making these changes, then don't. You can take the also not so great approach of simply copying your file share (all 5TB of it) into SharePoint to gain the functionality.

Once that is done, you should now be able to see your documents open in the embedded link generated:

So now that, that works. How do I get SharePoint Search to open the file as a preview? Hmmm…tricky. Let's look at how it does files that live in SharePoint. In order to do this, we have to open the search display templates for Office documents. The first one is for Word documents and it is called "Item_Word_HoverPanel.html" (yeah, they all have a different display template so you'll have to repeat the steps for each). You will notice some JavaScript that looks for a specific property ("ServerRedirectedEmbedURL"):

If this property exists (it is only populated by search if the content lives in SharePoint and is of a specific file type), then it will render the Office Web Apps preview area in the callout. In order to get OWA to work for fileshare files, we have to add the url that renders the link to the browser view or the embedded iframe. I wasn't able to get the iframe to work, but I did get a link that users can click on to get another browser to open with the file. You can do this by modifying the file to have the following:

<!–#_
        var i = 0;
        var wacurlExist = !Srch.U.e(ctx.CurrentItem.ServerRedirectedURL) && !Srch.U.e(ctx.CurrentItem.ServerRedirectedEmbedURL);
        var id = ctx.CurrentItem.csr_id;
        ctx.CurrentItem.csr_FileType = Srch.Res.file_Word;
        ctx.CurrentItem.csr_ShowFollowLink = true;
        ctx.CurrentItem.csr_ShowViewLibrary = true;
        ctx.currentItem_IsOfficeDocument = true;
        var find = '/';
       var re = new RegExp(find, 'g');

       function replaceAll(find, replace, str) {
return str.replace(new RegExp(find, 'g'), replace);
}

_#–>
        <div class="ms-srch-hover-innerContainer ms-srch-hover-wacSize" id="_#= $htmlEncode(id + HP.ids.inner) =#_">
            <div class="ms-srch-hover-arrowBorder" id="_#= $htmlEncode(id + HP.ids.arrowBorder) =#_"></div>
            <div class="ms-srch-hover-arrow" id="_#= $htmlEncode(id + HP.ids.arrow) =#_"></div>
            <div class="ms-srch-hover-content" id="_#= $htmlEncode(id + HP.ids.content) =#_" data-displaytemplate="WordHoverPanel">
                <div id="_#= $htmlEncode(id + HP.ids.header) =#_" class="ms-srch-hover-header">
                    _#= ctx.RenderHeader(ctx) =#_
                </div>
                <div id="_#= $htmlEncode(id + HP.ids.body) =#_" class="ms-srch-hover-body">
                <!–#_
                if ((ctx.CurrentItem.FileType == "docx") && Srch.U.n(ctx.CurrentItem.ServerRedirectedEmbedURL))
                {
                        ctx.CurrentItem.csr_DataShown = true;
                        ctx.currentItem_ShowChangedBySnippet = true;

                _#–>
<a href="https://svr-owa.contosocom/op/view.aspx?src=_#= $urlHtmlEncode(ctx.CurrentItem.Path.replace('file:','').replace(re,'%5C')) =#_" target="_blank">View File</a>

                <!–#_
                }
                _#–>

<!–#_
                    if(!Srch.U.n(ctx.CurrentItem.ServerRedirectedEmbedURL))
                    {
                        ctx.CurrentItem.csr_DataShown = true;
                        ctx.currentItem_ShowChangedBySnippet = true;
_#–>
                        <div class="ms-srch-hover-viewerContainer">
                            <iframe id="_#= $htmlEncode(id + HP.ids.viewer) =#_" src="_#= $urlHtmlEncode(ctx.CurrentItem.ServerRedirectedEmbedURL) =#_" scrolling="no" frameborder="0px" class="ms-srch-hover-viewer"></iframe>
                        </div>
                        <div class="ms-srch-hover-wacImageContainer">
                            <img id="_#= $htmlEncode(id + HP
.ids.preview) =#_" alt="_#= $htmlEncode(Srch.Res.item_Alt_Preview) =#_" onload="this.parentNode.style.display='block';" />
                        </div>
<!–#_
                    }
                    else
                    {
                        ctx.CurrentItem.csr_ShowLastModifiedTime = true;
                        ctx.CurrentItem.csr_ShowAuthors = true;
                    }

                    if(!Srch.U.e(ctx.CurrentItem.SectionNames))
                    {
                        ctx.CurrentItem.csr_DataShown = true;
_#–>
                        <div class="ms-srch-hover-subTitle"><h3 class="ms-soften">_#= $htmlEncode(Srch.Res.hp_SectionHeadings) =#_</h3></div>
<!–#_
                        var sectionNames = Srch.U.getArray(ctx.CurrentItem.SectionNames);

                        var sectionIndexes = Srch.U.getArray(ctx.CurrentItem.SectionIndexes);
                        if(!Srch.U.n(sectionIndexes) && sectionIndexes.length != sectionNames.length)
                        {
                            sectionIndexes = null;
                        }

                        var hitHighlightedSectionNames = Srch.U.getHighlightedProperty(id, ctx.CurrentItem, "sectionnames");
                        if(!Srch.U.n(hitHighlightedSectionNames) && hitHighlightedSectionNames.length != sectionNames.length)
                        {
                            hitHighlightedSectionNames = null;
                        }

                        var numberOfSectionsToDisplay = Math.min(Srch.SU.maxLinesForMultiValuedProperty, sectionNames.length);
                        var sectionsToDisplay = new Array();

                        var usingHitHighlightedSectionNames = Srch.SU.getSectionsForDisplay(
                            hitHighlightedSectionNames,
                            numberOfSectionsToDisplay,
                            sectionsToDisplay);

                        for(i = 0; i < sectionsToDisplay.length; ++i)
                        {
                            var index = sectionsToDisplay[i];
                            if(Srch.U.n(index))
                            {
                                continue;
                            }

                            var tooltipEncoded = $htmlEncode(sectionNames[index]);

This will get the "View File" link to display for the file. You can then click on it to have the file open in a new browser window and the users can now view the files!

Enjoy!
Chris

July 4, 2013

Content Type Hub publishing in mixed mode sites (14 vs 15) – Upgrade and migration planning

I noticed this post today from Brad Teed…It's a good one!:

http://sharepointsblog.com/2013/05/06/sharepoint-2013-content-type-hub/

I would have assumed that the content types would be the same in either mode, but evidentially not! There is a check in the code at Microsoft.SharePoint.Taxonomy.ContentTypeSync.Internal.PackageInfo.ValidatePackageVersion(). It checks the hub site collection compatibility level and then checks the current web's site collection compatibility level. If it doesn't match, then it errors out. This is a bummer for those that used the Content Publishing Hub in 2010 and must now upgrade all site collections at the same time to "15" in order to get updates from the hub.

However, there is something else interesting here. The APIs that are being used by the content type hub publishing are the Content Deployment APIs. So the package that it is checking is a simple export/import package that you would do with anything else. The thing that jumps out at me, that I would have assumed one could do, is export from 2010 and import into 2013 using those APIS, but it look like that is not a good path to take being that they specifically are denying you from doing this. This tells me that you should upgrade your older sites to 2013, then do an export, then do an import into 2013.

Chris

July 1, 2013

Create App Catalog Using Windows PowerShell

Here you go:

New-SPSite -Url http://teams.contoso.com/sites/AppCatalog -OwnerAlias "Contososp_admin" -Name "Contoso App Catalog" -Template "APPCATALOG#0"

Update-SPAppCatalogConfiguration -site http://teams.contoso.com/sites/AppCatalog

CJG

June 18, 2013

Office Web Apps callout disappears with "Allow management of content types"

When setting the following property:

You will lose your Office Web Apps callout:

Only to be replaced with this:

Being that just about every customer has added their own content types by now, and 3rd party tools add content types to document libraries, it makes the in-browser document creation feature of Office Web Apps a non-conversation piece. If you want to be able to create document in the browser, don't enable multiple content types on your document libraries!

CJG

May 22, 2013

Shredded Storage Testing Framework is POSTED!

You can download here:

http://shreddedstorage.codeplex.com

Whitepaper is still a few days out…

Enjoy!
Chris

May 14, 2013

The Most Interesting Guy in SharePoint

I have been pondering, what would the Dos Equis guy do in SharePoint land. This led to the following….

When building a 2013 Farm, he only needs….one server…

When he builds a SharePoint App, he doesn't need a secret key…
His SharePoint Apps, get a 6 rating from CJG
His SharePoint Azure Hosted Apps can be run on-premise

His Hybrid farm doesn't need….ADFS….

His claims identity….makes no claims about him…..

Microsoft instantly approves…his Apps….

His access tokens….don't expire….

He didn't need to look at Spence's post to setup Request Management….

His sandboxed solutions run with…full trust…

His SharePoint Groups…span site collections….

Every question he replied to on the MSDN forums….was marked as an answer…

When setting up authentication, he doesn't use setspn.exe….it just works…

His farm can have unlimited web applications…

Never once has he been asked an end user support question…..

His farm will never tell you "its working on it"….its just done….

He can delete terms from his MMS term store…

His SharePoint Online instance has….a Central Administration…

His SharePoint newsfeed…posts to Yammer….

His configuration database…scales out…

His farm can store files…..over 2GB….

He can post as many hashtags as he wants….in his newsfeed….

All SharePoint Farms trust his farm…..

He can unwind a shredded storage file with one line of code….

His machine translation service application is….perfect….

His Office Web Apps server, can edit PDFs in the browser…

His content databases….can store a petabyte of data….

He is….the most interesting guy in SharePoint…"I don't always collaborate…but when I do…its with SharePoint and hot chicks"…..

If you think of more and post, let me know…I'll link over!
CJG