Hybrid Search Results not displaying

Help!  My Hybrid Search isn't working!  What could it be?!?

  1. So you have successfully run the hybrid scripts here.
  2. You have ensured that the results are flowing to your Cloud Search Service Application via the Cloud SSA crawl logs.
  3. You go to do a search using your Azure AD cloud sync'd account and you get…nothing…what!??!
  4. You look things over and over again…maybe I didn't do this, maybe I didn't do that…no, it all looks good!

The possible cause:

  1. You didn't setup a User Profile Service Application (kinda rare I know)
  2. You view your "sync'd" site collection users profile and notice that the local "email address" does not match your cloud email address…doh!
    1. chris@companyone.com vs chris@companytwo.com
  3. You change the values to match using the edit list item feature of the site users list
  4. You re-run the cloud ssa crawl
  5. Go back to the cloud search center…ta da…results!  Just another reminded that UPNs have to match for the results to process and this value comes from the site collection users list when you have no UPS!

Hope this helps someone!
Chris

Other Helpful posts on a similar topic:

Cuba, Socialism vs Capitalism from a good ole boy – Recap of my trip to Cuba!

It's interesting to me that my last two trips that have been out of the country have been to China and Cuba, two communistsocialist countries.  Being I'm a USA born, good ole boy from Oklahoma, we can just say that it was as eye opening as moving from Oklahoma to Seattle and getting "hit on" the first time by a gay guy, which isn't bad, just different!

My Background: 

A bit of my background…I grew up with almost nothing, worked hard and the wonderful U. S. of A provided and rewarded me based on that hard work.  My mom worked hard and paid for everything from her job and the little child support dad gave us.  From what I remember, food stamps kicked in at one time, so to say I never drank from the US Democratic faucet would be false.  So, what does that have to do with anything?  Cuba.  What an eye opening experience, even more so than China as they seem to be some kind of mix of socialism and capitalism (unlike the pure socialism of Cuba).  As you will see, its not practical for someone to be able to out grow your neighbors salary wise as the average salary in Cuba is right around $40, with developers making somewhere around $500/month.  But the reality is they don't have to pay for their education, or their health care.  Imagine that…you can take as many courses as you want, get as smart as you want…or be as lazy as you want.  If only we could do that in the USA…

Getting there:

Getting to Cuba is very easy.  You can buy a Visa online before you go, or at the point of departure (the actual flight to Cuba) you can buy it.  You also need health insurance, although, being that health care is free, it was a bit odd at this was required. The price of your ticket currently includes $25 health insurance policy and your boarding pass serves as your insurance card.  Be sure that you don't lose the second part of the Visa on your way back!

Cuban Government: 

Most things in China are owned by the government, the hotel we stayed in for instance, the Hotel National, is owned by the government:

 

Although a nice hotel, it is old and run down.  There are other hotels that offer much nicer rooms and amenities, but to have stayed in a hotel that some of the greats have stayed in…we will just say it was an experience you can't compare to anything else!

 

We walked down to the US Embassy which is only about half a mile from the hotel.  Its on the water on the Malcon and its a very nice looking building:

 

Prices and Money: 

There are two currencies in Cuba.  The CUC and the CUP.  The CUC is for tourists, the CUP is for the locals.  You can pay with either if you have it, you just are likely to have CUCs.  I never had an issue with someone paying change in CUP back, but you should be aware of it and this blog is very helpful.  Net net is CUC are pictures of monuments, CUP are pics of individuals.  The price to change out the US dollar was ok at the airport.  The hotel was actually better by about .01.  On the way out, I got back 99% of my USD from my CUC at the airport.

The prices in Cuba are soooo cheap!  Most drinks (Mojito anyone?) are $5CUC at major hotels, if you venture out, you will find them for $2-3CUC and yes, they are super strong wherever you go!

 

Food:

The food in Cuba was amazing!  There was not a single meal that I didn't like!  The prices were soo cheap that I actually would buy 2 meals each time (yeah, I may look skinny, but I eat a lot!). By the way, a massive super yummy dinner for 9 of us at a sit down nice restaurant, was $130CUC.   The one thing you need to realize about sites like TripAdvisor is that the reviews are made by tourists that come off of cruise ships and its a part of their excursion package.  So if you try to go to one of these top 10 places, you will very likely be turned away as they book the entire venue for the excursion folks.  Face control seems to work well, so we were lucky and actually got into some of the places by smiling and laughing a bit…once we the staff realized we weren't with the cruise ships, we got much better attention!

 

People:

The people in Cuba were sooo nice!  You can tell that not having to worry about getting an education, getting treated for a disease or sickness made them very carefree and easy going.  As an aside, they created a vaccine for lung cancer, it makes sense as one of their major exports is tobacco!  I was able to bring back some Cohiba Behikes (52/54/56).  If you don't know what those are, you gotta look em up!

 

InternetNetwork: 

Just know that your hotel wifi will likely not be the same wifi as the public uses and the username/password you get for the RADIUS server won't work out on the open.  I had 45 minutes left on the hotel wifi and could not log into the ETECSA wifi at the airport! 

Socialism: 

So…get this…my presentation was focused on "how to make money" in Cuba. You can find it here.  Needless to say, making money in Cuba as a corporation is not something you will be able to do for quite some time.  Locals are not allowed to start corporations, so as a programmer, you can't start a consulting company and hire people.  The concept of a corporation doesn't really exist in Cuba.  You have to get government approval for everything and as we painfully learned, you also need permits for tech gathering! 

SharePointMicrosoftBusiness:

The people in Cuba do have a computer science based curriculum that teaches C#.  They use Microsoft products.  They don't have the best computers to run things on.  Most computers are imported via family and friends.  So most places aren't going to have some fancy server room where you can run the latest and greatest server OS and server based products.  You also can't count on the network to support cloud services so don't even think about selling cloud services there.  You have to have a local presence.  Someone will need to build a co-lo facility to host all the major players.  I'm sure it will be owned by the Cuban government.  You'll just have to put your servers in it to get any decent bandwidth.  Google is trying harder than all the others, but the progress is slow.  There is a trial to doing broadband.  I did see co-axel cables run all over, and there were lots of CAT5 cables run between houses…a lot of the cables were used for door bells though…LOL.

Soccer: 

Kids are playing soccer everywhere!  It was very hard for me to not actually get out and play (I have a torn ACL right now), but I did manage to get the courage to brave a fully tore ACL in front of a cool church (yeah those are my old Gucci soccer shoes):

 

Random Photos:

McFly!  McFly!  Yeah, there are old cars everywhere and at times the smog was a bit unbearable, this guy seemed to make the best of it:

 

Cubans love their country!  Some cools pics:

 

Get your drugs at a Harry Potter drug store!  This place was deep inside Old Havana in what we would easily call a "run down" part:

 

Our trip home:

Getting home is easy, the airport is pretty fast, although I can see if there were a lot of people it might be a bit crowded at times.  But we had no issues.  The only issue we had was the flight path back home, evidentially we flew over a tornado:

 

Summary: 

Although the major business opportunities in Cuba are in the low teens, and "USCapitalism" like careers non-existent, the environment is fun and the people are amazingly smart and carefree.  They live with what they have and make the best of it.  Something that a majority of Americans need to learn to appreciate.  As I came home and walked through my front door, I felt that appreciation for what I have (and what I don't have and have had), and all the opprotunities I have been given by being "born in the USA".

But at the same time, I felt really sorry for myself in having the wool pulled over my eyes with the stigma and residual of capitalism and how things really shouldn't be based solely on money and success, but on what you can contribute to your community, country and family.  It's what you don't know you don't know…

Go to Cuba before it changes too much!
Chris 

Using Nintex Workflow Cloud and Azure Functions to call on-premises workflows

This is from a lab from the soon to be released Nintex End User Training courses.  Its the first of its kind and I'll simply say, you will learn some fun stuff!  

  1. Login to
    Azure, or create a trial for Azure Functions
    • Trial
      https://functions.azure.com/try
    • Portal
      1. Select
        your subscription
      2. For
        the function name, type “NintexExternalStart”
      3. Select
        a region
      4. Click
        “Create + get started”, this can take a few seconds
  2. Click the
    “New Function” button
  3. Select the
    “HttpTrigger-CSharp” template:
  4. Click
    “Create"
 Copy the following to your azure function:
 
#r "Newtonsoft.Json"
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Security.Cryptography;
using System.Text;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, TraceWriter log)
{
    log.Info("C# HTTP trigger function processed a request.");    
    string url = "https://run.nintex.io/x-start/YourEndPoint";
    string securityKey = "YourSecurityKey";
    var variableValues = new Dictionary<string, object>();
    var service = new ExternalStartApiClient();
    try
    {
        log.Info("C# HTTP trigger function processed a request.");
        var correlationId = service.StartWorkflow(url, securityKey, variableValues, log);
    }
    catch
    {
    }
    string name = "Chris";
    return name == null
        ? req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a name on the query string or in the request body")
        : req.CreateResponse(HttpStatusCode.OK, "Hello " + name);
}
    
    public class ExternalStartApiClient
    {
        public RestClient Client { get; internal set; }
        #region Constructors
        public ExternalStartApiClient()
        {
            this.Client = new RestClient();
        }
        public ExternalStartApiClient(RestClient client)
        {
            this.Client = client;
        }
        #endregion
        /// <summary>
        /// Retrieves information about the workflow variables available for the Nintex workflow associated with the
        /// specified endpoint URL.
        /// </summary>
        /// <param name="endpointUrl">The External Start endpoint URL associated with a Nintex workflow.</param>
        /// <returns>A List object that contains a WorkflowVariable object for each available workflow variable.</returns>
        public List<WorkflowVariable> GetWorkflowVariables(string endpointUrl)
        {
            // Send a GET request to the specified endpoint URL. No authorization or authentication is
            // required for this request.
            HttpResponseMessage response = this.Client.Invoke(HttpMethod.Get, endpointUrl, null, null);
            // If a response is returned, check the HTTP status code.
            if (response != null)
            {
                switch (response.StatusCode)
                {
                    case HttpStatusCode.OK:
                        // Success – deserialize and return the list of workflow variables.
                        return this.Client.DeserializeJson<List<WorkflowVariable>>(response);
                    case HttpStatusCode.BadRequest:
                        // Failure – the endpoint URL could not access a workflow.
                        throw new ArgumentException("The request could not be processed.");
                    default:
                        throw new Exception("An unexpected error has occurred.");
                }
            }
            else
            {
                return null;
            }
        }
        /// <summary>
        /// Sends a message to start the Nintex workflow associated with the specified endpoint URL, optionally specifying 
        /// values for workflow variables defined by that workflow
        /// </summary>
        /// <param name="endpointUrl">
        /// The External Start endpoint URL associated with a Nintex workflow.
        /// </param>
        /// <param name="securityKey">
        /// The security key associated with the endpoint URL.
        /// </param>
        /// <param name="workflowVariables">
        /// The workflow variable values to use when starting the workflow.
        /// </param>
        /// <returns>
        /// The <see cref="string"/>.
        /// The X-CorrelationId: {GUID} of the request
        /// </returns>
        public string StartWorkflow(string endpointUrl, string securityKey, Dictionary<string, object> workflowVariables, TraceWriter log)
        {
            // If no workflow variable values are provided, send an empty request body; otherwise, send
            // a serialized JSON object, in which each workflow variable value is represented as a property value.
            string requestBody = "";
            if (workflowVariables != null)
            {
                JObject associationData = new JObject();
                foreach (string key in workflowVariables.Keys)
                {
                    JProperty value = new JProperty(key, workflowVariables[key]);
                    associationData.Add(value);
                }
                requestBody = associationData.ToString();
            }
            // Retrieve and configure the values used to calculate the digest value for the request.
            var path = new Uri(endpointUrl).AbsolutePath.ToLower();
            var httpMethod = HttpMethod.Post.ToString().ToLower();
            var nonce = Guid.NewGuid().ToString();
            var timestamp = DateTime.UtcNow.ToString("O");
            // Calculate and return the digest value for the request.
            var digest = CalculateDigest(securityKey, httpMethod, path, nonce, timestamp, requestBody);
            log.Info(digest);
            log.Info(timestamp);
            log.Info(nonce);
            // Specify the header values for the request.
            var headerValues = new Dictionary<string, string>();
            headerValues.Add("X-Api-Digest", digest);
            headerValues.Add("X-Api-Timestamp", timestamp);
            headerValues.Add("X-Api-Nonce", nonce);
            headerValues.Add("X-Api-Source", "ExternalStart");
            // Send the request to the endpoint URL. 
            HttpResponseMessage response = this.Client.Invoke(HttpMethod.Post, endpointUrl, headerValues,
                new StringContent(requestBody, Encoding.UTF8, "application/json"));
            
            if (response != null)
            {
                log.Info(response.StatusCode.ToString());
                switch (response.StatusCode)
                {
                    case HttpStatusCode.OK:
                        // Success – the message was successfully sent.
                        IEnumerable<string> correlationIds;
                        response.Headers.TryGetValues("X-CorrelationId", out correlationIds);
                        return correlationIds.FirstOrDefault();
                        break;
                    case HttpStatusCode.BadRequest:
                        // Failure – the endpoint URL could not access a workflow.
                        throw new ArgumentException("The request could not be processed.");
                    default:
                        throw new Exception("An unexpected error has occurred.");
                }
            }
            else
            {
                throw new Exception("An unexpected error has occurred.");
            }
        }
        /// <summary>
        /// Calculate the digest value used to authenticate requests for the External Start feature.
        /// </summary>
        /// <param name="securityKey">The security key associated with an External Start endpoint URL.</param>
        /// <param name="httpMethod">The HTTP method name, in lower case.</param>
        /// <param name="path">The absolute URI of the External Start endpoint URL, in lower case.</param>
        /// <param name="nonce">The nonce value.</param>
        /// <param name="timestamp">The date and time of the request, in ISO 8601 format.</param>
        /// <param name="requestBody">The serialized body of the request.</param>
        /// <returns>A keyed hash value, using the SHA-256 function, to be used as the Hash-based Authentication Code (HMAC) value for a request.</returns>
        public string CalculateDigest(string securityKey, string httpMethod, string path, string nonce,
            string timestamp, string requestBody)
        {
            // The data values are concatenated into a single string, in which each data value is delimited by 
            // a colon (:) character, which is then encoded as a UTF-8 byte array.
            var dataBytes = Encoding.UTF8.GetBytes(String.Join(":", httpMethod, path, nonce, timestamp, requestBody));
            // The security key is encoded as a UTF-8 byte array.
            var keyBytes = Encoding.UTF8.GetBytes(securityKey);
            // Using the HMACSHA256 object provided by .NET Framework, the
            // data values are hashed, using the security key, and any dashes are removed.
            using (var hasher = new HMACSHA256(keyBytes))
            {
                var hashBytes = hasher.ComputeHash(dataBytes);
                return BitConverter.ToString(hashBytes).Replace("-", "");
            }

        }
    }
    public class RestClient
    {
        public NetworkCredential Credential { get; set; }
        #region Constructors
        public RestClient()
        {
        }
        public RestClient(NetworkCredential credential)
        {
            this.Credential = credential;
        }
        #endregion
        #region Deserialization
        /// <summary>
        /// Deserialize the body of a specified HTTP response as an instance of a specified type.
        /// </summary>
        /// <typeparam name="T">The type into which to deserialize.</typeparam>
        /// <param name="restResponse">The HTTP response from which to deserialize.</param>
        /// <param name="jsonConverters">If needed, any custom JSON converters with which to deserialize.</param>
        /// <returns>An instance of the specified type, deserialized from the body of the specified HTTP response.</returns>
        internal T DeserializeJson<T>(HttpResponseMessage restResponse, JsonConverter[] jsonConverters = null)
        {
            return this.DeserializeJson<T>(restResponse.Content.ReadAsStringAsync().Result, jsonConverters);
        }
        /// <summary>
        /// Deserialize a specified JSON-encoded string as an instance of a specified type.
        /// </summary>
        /// <typeparam name="T">The type into which to deserialize.</typeparam>
        /// <param name="jsonString">The string from which to deserialize.</param>
        /// <param name="jsonConverters">If needed, any custom JSON converters with which to deserialize.</param>
        /// <returns>An instance of the specified type, deserialized from the body of the specified string.</returns>
        internal T DeserializeJson<T>(string jsonString, JsonConverter[] jsonConverters = null)
        {
            if (jsonConverters != null)
                return JsonConvert.DeserializeObject<T>(jsonString, jsonConverters);
            else
                return JsonConvert.DeserializeObject<T>(jsonString);
        }
        #endregion
        #region Serialization
        /// <summary>
        /// Serialize the specified object as a JSON-encoded string.
        /// </summary>
        /// <param name="value">The object from which to serialize.</param>
        /// <param name="jsonConverters">If needed, any custom JSON converters with which to serialize.</param>
        /// <returns>A JSON-encoded string that contains the serialization of the specified object.</returns>
        internal string SerializeJson(object value, JsonConverter[] jsonConverters = null)
        {
            if (jsonConverters != null)
            {
                return JsonConvert.SerializeObject(value, Formatting.Indented, jsonConverters);
            }
            return JsonConvert.SerializeObject(value, Formatting.Indented);
        }
        #endregion
        #region REST invocation
        /// <summary>
        /// Invokes the specified REST resource, using the specified HTTP method, optionally providing any specified header values and request content.
        /// </summary>
        /// <param name="operationMethod">The HTTP method with which to invoke the REST resource.</param>
        /// <param name="operationUrl">The operation URL with which to invoke the REST resource.</param>
        /// <param name="operationHeaders">A collection of header names and values to include when invoking the REST resource.</param>
        /// <param name="operationContent">The HTTP content to include when invoking the REST resource.</param>
        /// <returns></returns>
        internal HttpResponseMessage Invoke(HttpMethod operationMethod, 
            string operationUrl, 
            Dictionary<string, string> operationHeaders, 
            HttpContent operationContent)
        {
            HttpResponseMessage response = null;
            try
            {
                // Instantiate a new HttpClientHandler object and, if credentials are provided,
                // configure and include them.
                var clientHandler = new HttpClientHandler { PreAuthenticate = true };
                if (this.Credential == null)
                {
                    clientHandler.UseDefaultCredentials = true;
                }
                else
                {
                    clientHandler.Credentials = this.Credential;
                }
                // Instantiate a new HttpRequestMessage, using the specified HTTP method and operation URL,
                // for the request.
    &n
bsp;           var request = new HttpRequestMessage(operationMethod, operationUrl);
                // If header values are provided, add them to the request.
                // NOTE: The implementation is not optimal, but suffices for the sample.
                if (operationHeaders != null)
                {
                    foreach (var key in operationHeaders.Keys)
                    {
                        request.Headers.Add(key, operationHeaders[key]);
                    }
                }
                if (operationContent != null)
                {
                    request.Content = operationContent;
                }
                // Instantiate a new HttpClient, asynchronously invoke the REST resource,
                // and await the result. 
                using (var client = new HttpClient(clientHandler))
                {
                    response = client.SendAsync(request).GetAwaiter().GetResult();
                }
            }
            catch (Exception)
            {
                response = null;
            }
            // Return the resulting HttpResponseMessage.
            return response;
        }
        #endregion
    }
    public enum VariableTypes
    {
        Text,
        TextMultipleLine,
        Number,
        DateTime,
        Boolean
    }
    /// <summary>
    /// Models the information returned for workflow variables by the External Start feature.
    /// This class is used to deserialize that information for the purposes of the sample.
    /// </summary>
    public class WorkflowVariable
    {
        [JsonProperty("Name"), JsonRequired]
        public string Name { get; internal set; }
        [JsonProperty("Type"), JsonRequired]
        public VariableTypes Type { get; internal set; }
        [JsonProperty("Required"), JsonRequired]
        public bool Required { get; internal set; }
    }
 
 And then you'll need a project.json file:
 
 {
  "frameworks": {
    "net46":{
      "dependencies": {
        "Newtonsoft.Json": "8.0.3"
      }
    }
   }
}

Nintex Training (Advanced) – Its almost here!

New Nintex Courses are coming! 

Yup, Nintex is such a great product and it has been making so many strides in so many ways, I decided to go ahead and build a set of in-depth courses that cover just about everything you could possibly imagine about Nintex on-premises.  Currently the Nintex customer base is focused on 2013, but many people are moving to 2016 and Office 365 so another set of courses is coming in mid-2017 to cover those technologies.  The courses will become officially available in mid-February 2017, but I would love to have some testers that would like to get up to speed on Nintex quickly and get some free training if you give me feedback to cycle back in!

Cloud Services 

As some of you may be aware, Nintex has started to move its technology investments to cloud based applications.  The new Nintex Hawkeye, Nintex Workflow Cloud (competitor to PowerApps) and other items in the pipeline are designed to leverage the benefites of the cloud.  The new courses cover those technologies too (a first for Nintex courses)!

Nintex Subscription Pricing 

In addition, ACS has become a Nintex Partner so we can sell you Nintex subscriptions.  I'm not greedy so you can imagine the pricing will be pretty amazing, just drop me a line and I'll get you a quote!  Not only that, but I'll even throw in free training courses with the purchase in most cases!

You can find out more about the courses here:

http://www.architectingconnectedsystems.com/Nintex.aspx

Enjoy! 
Chris 

PowerStreamECM and Extreme Mode!

Fun, cool announcement to make.  

PowerStreamECM, a tool I have built for large scale ECM legacy migrations now supports EXTREME Mode for Office 365!

Extreme mode takes advantage of the Azure to O365 path using the content deployment API methodologies of SharePoint.  The fun interesting thing about Extreme mode is its designed to work with hard drive based deployments to Azure.  Sending 1GB, 10GB, 100GB to Azure is ok…try sending 30TB, or even 100TB to Azure over standard network lines….it just doesn't work.  Extreme mode allows you to partition your data across multiple drives and then send those drives to Azure for upload.  The entire process takes about 2-4 weeks.  

Once the data is loaded, you can then fire a secondary process to upload and modify any custom properties based on custom business rules to update the pre-loaded binaries.

Once uploaded, custom and 3rd party tools can apply the retention rules you are looking for to destroy old records or do eDiscovery.  Unfortunately, the only thing about uploading 30-100TB of data is that O365 can't handle it well for certain features.  That's where the experience of Hyper-scale design comes into play.  We have built a set of rules and software that allows you to do hyper-scale O365 deployments easily (although, time always plays a factor in hyperscale, just look at big data)!

If you have an old ECM system that has over 30TB of data that you want to move to O365, give me a shout.  I have seen it all at this point and can tell you what works and what doesn't!

Enjoy!
Chris 

Nintex XChange Asset Review

Nintex earlier this year unveiled a community portal for community members to provide helpful contributions to the Nintex user base called Nintex XChange.  It follows along with the "App marketplace" style of the Office 365 app store. 

To date there are 80 contributions, here are my ratings (5 highest) on all of them with some extra data points at the end:

AssetId Name Comment Rating DeploymentType AssetType Platform
3686 Uber Price Estimate UDA Uber price estimate UDA…sweet! 5 Workflow Template;UDA UDA Nintex Workflow
3947 Multi-Step form for Office 365 An example of how to make a Nintex Form that is multi stepped. This is pretty freakin awesome! 5 JavaScript JavaScript Nintex Forms
3968 Data One 4 PowerShell Woah…running Server Side OM in a Nintex Action…Cool! Comes with intellsense and everything…dang! 5 SharePoint Solution Custom Action Nintex Workflow
3949 Parsing XE.com data – UDA Currency convertor UDA, nice! 4 UDA UDA Nintex Workflows
4210 How to Create Cascading Choice Controls – Nintex Forms A mix of CSS and JavaScript to implement cascading drop downs. Good Stuff! 4 CSS; JavaScript CSS; JavaScript Nintex Forms
4237 Extract Lookup ID A regular expression to get the item ID from a lookup control. 4 JavaScript JavaScript Nintex Forms
5896 Wunderlist UDAs for Nintex Workflow On-Prem A UDA that wraps Wunderlist API REST Calls. Actually made me go sign up so I could check it out…interesting! 4 UDA User Defined Action Nintex Workflow
3894 Nintex Add-in for Outlook Nintex Add in for outlook. A simple app that looks for links in an email to notify a person that it is a Lazy Approval task. 4 Add-In Add-In Nintex Workflow
3943 Visual Studio Post Deployment Script – ALCM A VisualStudio Post Deployment script for deploying your Nintex Artifacts to SharePoint from Visual Studio. Very handy! 4 PowerShell PowerShell Windows
3896 Show Repeating Section as table in List View A script to show a repeating section as table in SharePoint View 4 JavaScript JavaScript Nintex Forms
3897 Custom Validation for Nintex Forms An exmaple of how to write your own JavaScript custom validator. Very helpful! 4 JavaScript JavaScript Nintex Forms
3903 Generate an HTML document with dynamic data Interseting take on generating a document from metadata in HTML format 4 Workflow Template Workflow Nintex Workflow
3923 Iterate through documents in a Document Set Workflow that iterates through all the files in a document set. Coolness… 4 Workflow Template Workflow Nintex Workflow
3925 Archiving a List Item with Workflow History A UDA that backs up workflow history from one instance to another. Interesting pattern. 4
3936 Terminate Old Instances of a Workflow A UDA to terminate old workflow instances. Interesting… 4 UDA UDA Nintex Workflow
3937 Business Days Validation for Forms Business day calculation JavaScript. 4 JavaScript JavaScript Nintex Forms
3939 Convert Forms to PDF w/ Workflow Convert a form to a PDF using workflow. Pretty sweet use of out of box technology. 4 Workflow Template Workflow Nintex Workflow; Nintex Forms
3706 Sending a TXT w/ Nintex Workflow How to send a SMS with Twilio via a web request – nice! 4 Guide Guide Nintex Workflow
3723 Latitude and Longitude UDA How to get Latitude and Longitude via web request – nice solution 4 UDA UDA Nintex Workflow
3607 Start Workflow with PowerShell Useful powershell script to start a nintex workflow 4 PowerShell PowerShell Nintex Workflow
3729 Resetting Repeating Sections A custom action to reset the items in a repeating section. This is actually pretty darn handy! 4 SharePoint Solution Custom Action Nintex Workflow
3841 Using the SharePoint REST service inside Nintex Forms An example of how to query SP REST endpoints using JavaScript 4 JavaScript JavaScript All
3876 Tab-based Layout Solution How to make your forms tab based. Very helpful! 4 CSS CSS Nintex Forms
3891 Embedding Signature Pad using JSignature.min.js An example of how to use JavaScript to embed a signature pad into your Nintex Forms. 4 JavaScript JavaScript Nintex Forms
3892 Splitting Text – RegEx Example of a regex usage in Nintex Workflow 3 Guide Guide Nintex Workflow
3858 Parsing Nintex Forms Repeating Section Solution – O365 A second type of master-child form using workflow to break out the items. This one is for Office 365. Similar to 3385. 3 Workflow Template Workflow Nintex O365
3859 Get Recurring Events – UDA UDA to get recurring events…not bad… 3 Workflow Template Workflow Nintex Workflow
3846 Dynamically Generated Hyperlinks JavaScript to generate hyperlinks to document in a Form and in O365 3 JavaScript JavaScript Nintex O365
3847 Create Salesforce Lead via Nintex Workflow Creates a SalesForce lead via Nintex Workflow 3 Guide Guide Nintex Workflow
3848 Dynamic approvers for douments Workflow for doing dynamic approvals on a document. Another basic pattern, but helpful if you haven't seen the pattern before. 3 Workflow Template Workflow Nintex Workflow
3758 Nintex Form File Attachment Validation Settings Nintex form that only allows an upload of PDF files 3 Nintex Form Nintex Form Nintex Forms
3385 Parent/Child Forms/Workflow Solution An example of how to use the repeating section of a nintex form to drive list item creation via workflow. Pretty basic pattern, but useful if you haven't done it before. 3 Nintex Forms
3725 Set image based on location in Forms Set an image based on GeoLocation 3 JavaScript JavaScript Nintex Forms
3726 Start a workflow & wait w/ Form JavaScript to start a workflow from Nintex Forms. Interesting implementation… 3 JavaScript JavaScript Nintex Forms
3728 SAP Made Easy w/ Zimt SAP Actions from Zimt 3 SharePoint Solution Custom Actions Nintex Workflows
3608 PowerShell Find All Workflows – Part 1 Shows how to get all the Nintex workflows via the NWAdmin tool. More for reporting than anything else. 3 PowerShell PowerShell Windows
3613 Safely purge items from history list – PowerShell PowerShell script to purge the workflow history list. 3 PowerShell PowerShell Nintex Workflow
3940 Add a Site Workflow Link in O365 A way to add a link to Nintex Site Workflows to quick nav 3 Other Other SharePoint
3895 Replace Text – RegEx An example of using regex to do a replacement 3 RegEx RegEx Nintex Workflow
4073 Layer2 Cloud Connector A data connector for Nintex Workflow. Doesn't really give any helpful patterns of why you should buy/use it. 3 SharePoint Solution Custom Action Nintex Workflow
4137 Theobald Software – ERPConnect Services  Theobald Software – ERPConnect Services 3 SharePoint Solution Custom Actions Nintex Workflow
4017 Delete Attachments – UDA UDA to delete attachments 3 UDA UDA Nintex Workflow
4018 What Changed – UDA UDA for tracking what changed in an item 3 UDA UDA Nintex Workflow
4019 Random Number – UDA A UDA that will generate a random number 3 UDA User Defined Action Nintex Workflow
4060 Panels UI Magic with Nintex Forms for Office 365 Another example of how to do a "Stepped" or "Paneled" nintex Form 3 CSS CSS Nintex Forms
4153 Action Task Reminder – Site Workflow Another pattern around task reminders. 3 Workflow Template Workflow Nintex Workflow
4155 Dynamic button label Dynamic button label via calculated value and events 3 JavaScript JavaScript Nintex Forms
4157 Remove Permissions UDA UDA to remove permission down to read only 3 UDA UDA Nintex Workflow
4174 Set Manager in a Nintex Forms People Picker Control A javascript code snippet (must be paid for), to populate the manager of a people form control. I'm betting no one has really paid for this… 2 JavaScript JavaScript Nintex Forms
4063 Simple email validation for forms Basic email reg ex. 2 Regular Expression Text Nintex Forms
4147 Document Review and Approval (State Machine) Workflow An example of a workflow to do review and approvals. 2 Workflow Template Workflow Nintex Workflow
3950 Document Review Reminder Process Basic reminder pattern. 2 Workflow Template Workflow Nintex Workflow
5974 WYSIWYG Business Cards in Office 365  SharePoint  Online using @Nintex Forms and Workflow apps for @Office365 A video of how to build a business card application with preview capabilites 2 Video Video Nintex Forms
6071 Trigger a Nintex Workflow from DropBox w/ Zapier Just a walkthrough of connecting your external start to Nintex Workflow for DropBox/Zapier. 2 Guide Guide Nintex Workflow
6072 Trigger a Nintex Workflow from OneDrive w/ Zapier Just a walkthrough of connecting your external start to Nintex Workflow for OneDrive/Zapier 2 Guide Guide Nintex Workflow
6073 Trigger a Nintex Workflow from Salesforce w/ Zapier Just a walkthrough of connecting your external start to Nintex Workflow for SalesForce/Zapier 2 Guide Guide Nintex Workflow
6082 Trigger a Nintex Workflow from Zendesk w/ Zapier Just a walkthrough of connecting your external start to Nintex Workflow for Zendesk/Zapier. 2 Guide Guide Nintex Workflow
3941 Upload ULS Logs to Doc Library – PowerShell A powershell script to upload ULS logs. Kinda odd, but I guess? 2 PowerShell PowerShell Windows
3724 Hide Form Footer How to hide the Nintex Form footer 2 CSS CSS Nintex Forms
3772 Birthday UDA A UDA that figures out the day of the week you were born on. Fun, but not very useful. 2 UDA UDA Nintex Workflow
3797 Replace Empty String Variable A UDA that will replace an empty string with something else. Hmmm… 2 UDA UDA Nintex Workflow
3798 Site Creation Workflow Workflow to create a site. 2 Workflow Template Workflow Nintex Workflow
3820 Set Placeholder Text Hints for Text Box Controls Just some javascript to show sample text in a text box on your Nintex Form. Pretty basic. 2 JavaScript JavaScript Nintex Forms
3853 Create user group in your SharePoint Site – UDA A user defined function (UDF) for Nintex 2010 that allows you to create a SharePoint groups 2 UDA User Defined Action Nintex Workflow
3854 Remove user group from SharePoint Site – UDA A user defined function (UDF) for Nintex 2010 that allows you to removew users from SharePoint groups 2 UDA User Defined Action Nintex Workflow
3855 Add user to a SharePoint group – UDA A user defined function (UDF) for Nintex 2010 that allows you to add users to SharePoint groups 2 UDA User Defined Action Nintex Workflow
3856 Remove user from SharePoint group – UDA Remove a user from SP Group 2 UDA UDA Nintex Workflow
3857 Math Power – UDA Math Power UDA…funny comments… 2 UDA UDA Nintex Workflow
3893 Extract Text – RegEx An example of how to run a regular expressions against some text. Not very helpful. 2 Other Text Nintex Workflow
3887 Sample Case Summary Template – Salesforce SalesForce case study doc…for DrawLoop stuff… 2 Guide Guide Drawloop
3888 Sample Cover Letter Template – Drawloop SalesForce case study doc…for DrawLoop stuff… 2 Guide Guide Drawloop
3889 Sample Quote Template – Drawloop SalesForce case study doc…for DrawLoop stuff… 2 Guide Guide Drawloop
3590 Theobald SAP – Connector Theobald SAP – Connector – looks like it has absoltely nothing valuable to do… 1 ? ? ?
3614 <Xchange Template – Delete this field and replace with your title> Hmm…nada here… 1 ? ? ?
3948 Expense Report Solution – PointBeyond supposed to be an expense form, but nothing here 🙁 1 Other Other Nintex Forms
3944 Build a Nintex form – pt 1 of 3 Simple tutorial – should probably not be in the XChange 1 Guide Guide Nintex Forms
3945 Build a Nintex form – pt 2 of 3 Simple tutorial – should probably not be in the XChange 1 Guide Guide Nintex Forms
3946 Build a Nintex form – pt 3 of 3 Simple tutorial – should probably not be in the XChange 1 Guide Guide Nintex Forms
6111 SharePlus & Nintex Integrated Solution : Introduction Nothing but a 3rd party adverisement/marketing slick 1 Guide Guide Nintex Workflow;Nintex Forms
4209 Nintextionary – Nintex Actions Dictionary Have no idea what this does or why it is posted. Doesn't seem to have any assets at all! 1 Other Other Nintex Workflow

Platforms:

  • 44 Nintex Workflow
  • 25 Nintex Forms
  • 3 Drawloop
  • 3 Windows
  • 2 ?
  • 2 Nintex O365
  • 1 All
  • 1 SharePoint

Deployment Types:

  • 15    UDA
  • 14    Guide
  • 13    JavaScript
  • 10    Workflow Template
  • 5    SharePoint Solution
  • 5    PowerShell
  • 4    Other
  • 3    CSS
  • 2    NULL
  • 2    ?
  • 1    Add-In
  • 1    CSS; JavaScript
  • 1    RegEx
  • 1    Regular Expression
  • 1    Nintex Form
  • 1    Video
  • 1    Workflow Template;UDA

Top Views:

  • 3771 – Convert Forms to PDF w/ Workflow
  • 2330 – Terminate Old Instances of a Workflow
  • 2135 – Nintex Add-in for Outlook
  • 2055 – Parsing Nintex Forms Repeating Section Solution – O365
  • 2018 – Set Placeholder Text Hints for Text Box Controls
  • 2009 – PowerShell Find All Workflows – Part 1
  • 1915 – Document Review Reminder Process
  • 1906 – Build a Nintex form – pt 1 of 3
  • 1868 – Custom Validation for Nintex Forms
  • 1855 – Tab-based Layout Solution
  • 1810 – Iterate through documents in a Document Set
  • 1808 – Using the SharePoint REST service inside Nintex Forms
  • 1800 – Splitting Text – RegEx
  • 1800 – Parent/Child Forms/Workflow Solution
  • 1734 – Start Workflow with PowerShell 

Top Likes:

  • 17 – Nintex Add-in for Outlook
  • 11 – Show Repeating Section as table in List View
  • 11 – Convert Forms to PDF w/ Workflow
  • 11 – Document Review Reminder Process
  • 11 – Simple email validation for forms
  • 11 – Tab-based Layout Solution
  • 10 – Set Placeholder Text Hints for Text Box Controls
  • 10 – Generate an HTML document with dynamic data
  • 10 – Iterate through documents in a Document Set
  • 9 – Business Days Validation for Forms
  • 9 – Using the SharePoint REST service inside Nintex Forms
  • 8 – Embedding Signature Pad using JSignature.min.js
  • 8 – Parent/Child Forms/Workflow Solution
  • 8 – Build a Nintex form – pt 1 of 3
  • 8 – Multi-Step form for Office 365 

Top Contributors:

  • 54 Eric Harris
  • 5 Patrick Abel
  • 2 rbachmann
  • 2 Jesse McHargue
  • 2 Swetha Sankaran
  • 2 Brad Orluk
  • 2 Cassy Freeman
  • 1 Chris Sullivan
  • 1 Christian Tauchmann
  • 1 Dean Virag
  • 1 Frank Daske
  • 1 frankgr
  • 1 jackgelo
  • 1 Jan Eyres
  • 1 Jay Cheong
  • 1 Vadim Tabakman
  • 1 Andrew Glasser
  • 1 Samantha Pugh

 

Data Loss Prevention (DLP) Architecture – Deep Dive

One of the hot features of Office 365 and "by inheritance" in SharePoint 2016 is Data Loss Prevention (DLP).  DLP allows you to execute search queries on your content to find possible "Sensitive Data Types".  You can find many blog post on how to setup DLP queries and how to find data, but that is at the high level.  In this blog post, I get into the nitty gritty of how it actually works under the covers.

High Level Steps:

  • UserApp uploads a document
  • Search gather runs (likely every 15 minutes)
  • Search content indexes the content
    • Ceres engine executes flows, one such Flow is called "Data Loss Prevention"
      • DLP Flow runs
      • DLP Flow looks for keywords
      • DLP Flow looks for regular expression matches
      • DLP excludes invalid "test" values
      • DLP Flow adds "SensitiveType" crawl property data
  • Ceres adds data to the index
  • User queries for DLP data using the "SensitiveType" managed proprerty

Deep Level Steps:

Most of the major work happens in the Ceres Flow.  It uses a "configuration" file of sorts from the "Microsoft.Ceres.DataLossPrevention" .NET Assembly in the GAC.  You will find a "defaultCLassificationRules" resource file in this assembly.  This file is called a "RulePackage" and contains Rule Packs. Currently for SharePoint 2016, there is only one rule pack, Office 365 may have more. Opening this file you will find the following:

  • Entities
    • High level container of a particular "end user" data type
      • US Driver's License Number – there are different in almost every state
      • Canada Driver's License Number – different in each of the provinces
    • Made up of patterns
  • Patterns
    • Made up of IDMatch'es, and Keyword Matches
    • Patterns have confidence levels.  Most expect a confidence level of 75% or higher.
    • Pattern can enforce minMatches (must have at least one keyword, etc)
  • RegEx Expressions
  • Keywords
    • Keywords may be case sensitive – most are not
  • Localization strings – these are used for the rendering and matching of keywords

Patterns:

  • Credit Card Number
  • EU Debit Card Number
  • US Social Security Number
  • US Individual Taxpayer Id Number (ITIN)
  • Canada Social Insurance Number
  • UK NINO
  • UK Driver's License
  • German Driver's License Number
  • German Passport Number
  • UK NHS Number
  • France INSEE
  • France Driver's License
  • Canada Driver's License
  • US Driver's License
  • Japan Driver's License
  • Japan Resident Registration
  • Japan Social Insurance Number
  • Japan Passport Number
  • Japan Bank Account Number
  • France Passport Number
  • US/ UK Passport Number
  • SWIFT Code
  • US Bank Account Number
  • ABA Routing Number
  • DEA Number
  • Australia Medical Account Number
  • Australia Tax File Number
  • Israel National ID Number
  • New Zealand Health Number
  • Spain SSN
  • Sweden National ID
  • Australia Bank Account Number
  • Australia Passport Number
  • Canada Bank Account Number
  • Canada Passport Number
  • Canada PHIN
  • Canada Health Service Number
  • France CNI
  • IP Address
  • IBAN
  • Israel Bank Account Number
  • Italy Driver's license Number
  • Saudi Arabia National ID
  • Sweden Passport Number
  • U.K. Electoral Number
  • Finnish National ID
  • Taiwanese National ID
  • Poland National ID (PESEL)
  • Poland Identity Card
  • Poland Passport Number

Regular Expressions:

Here are some of the regular expressions that DLP flow is looking for:

  • <Regex id="Regex_france_cni">(^|s)(d{12})($|s|.s)</Regex>
  • <Regex id="Regex_uk_electoral">(^|s)([a-zA-Z]{2}d{1,4})($|s|.s)</Regex>
  • <Regex id="Regex_canada_health_service_number">(^|s)(d{10})($|s|.s)</Regex>
  • <Regex id="Regex_canada_phin">(^|s)(d{9})($|s|.s)</Regex>
  • <Regex id="Regex_canada_passport_number">(^|s)(D{2})(d{6})($|s|.s)</Regex>
  • <Regex id="Regex_canada_bank_account_number">(^|s)(d{7})($|s|.s)</Regex>
  • <Regex id="Regex_australia_passport_number">(^|s)([A-Za-z]d{7})($|s|.s)</Regex>
  • <Regex id="Regex_australia_drivers_license_number">(?ix)(?:^|s)(?=(?:[A-Zd]{2}d{2}[A-Zd]{5})(?:$|s|.s))(?=(?:[A-Z]{0,2}d){4,9})(?!(?:d{0,9}[A-Z]){3,9})[A-Zd]{9}</Regex>
  • <Regex id="Regex_australia_bank_account_number">(^|s)([0-9]{6,10})($|s|.s)</Regex>
  • <Regex id="Regex_sweden_passport_number">(^|s)(d{8})($|s|.s)</Regex>
  • <Regex id="Regex_italy_drivers_license_number">(^|s)(D{1}[^b-uw-zB-UW-Z])((w{7})(D))($|s|.s)</Regex>
  • <Regex id="Regex_ipv4_address">(^|s)((?:[0-9].)|(?:[0-9]))(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(?!(?:.[0-9])|(?:[0-9]))($|s|.s)</Regex>
  • <Regex id="Regex_ipv6_address">(^|s)((?:[A-F0-9]{1,4}:){7}[A-F0-9]{1,4})($|s|.s)</Regex>
  • <Regex id="Regex_israel_bank_account_number">(^|s)(d{2}-d{3}-d{8}|d{13})($|s|.s)</Regex>
  • <Regex id="Regex_saudi_arabia_national_id">(^|s)(d{10})($|s|.s)</Regex>
  • <Regex id="Regex_usa_bank_account_number">(^|s)(d{4,17})($|s|.s)</Regex>

Keywords:

Keywords are important for deciding the confidence level of a particular pattern.  If a keyword is found, then the confidence goes up.  Here are some examples:

  •  UK Nino contains:
    • national insurance number,national insurance contributions,protection act,insurance,social security number,insurance application,medical application,social insurance,medical attention,social security,great britain,insurance
  • UK Driver's License:
    • DVLA,light vans,quadbikes,motor cars,125cc,sidecar,tricycles,motorcycles,photocard licence,learner drivers,licence holder,licence holders,driving licences,driving licence,dual control car

Keep in mind that your SharePoint on-premises environment will always have a static rule pack unless you download and apply updates.  Office 365 will continually get updated rule packs to enhance the DLP engine features so you will be more protected against sensitive data "leaks".

Enjoy!
Chris

Introducing the Microsoft MVP Bot!

At the #MVPSummit I sat in a couple of Microsoft Bot Framework sessions.  Cool stuff for sure!  So I decided to write a helpful program for the MVP Program that lets you look up MVP data from your favorite messaging platforms!  

Check it out via Consumer Skype here:

https://join.skype.com/bot/6f9b5f20-1a17-4b53-b7b9-f5072e0fe44e

Microsoft Teams:

https://teams.microsoft.com/l/chat/0/0?users=28:6f9b5f20-1a17-4b53-b7b9-f5072e0fe44e

Kik:

https://bots.kik.com/#/mvplookup 

Phrases you can use:

  • "Hi", "Hello"
  • What MVP Categories are there? 
  • Is Chris Givens an MVP?
  • Contact Chris Givens
  • MVPs in Norway
  • What does vlad catrinescu look like?
  • Chris Givens Picture
  • Chris Givens Profile
  • Search michele
  • Oldest
  • Who are 5 year mvps?
  • Who are Microsoft Azure mvps?

Enjoy!
Chris 

 

 

Follow every Microsoft MVP on Twitter!

Updated as of 11/20/2016 – Several MVPS updated their profiles to add their twitter handle or make it public. 

Updated as of 11/5/2016 – Re-posted the list creation to keep list name under limit.  Added the ability for you to clear out old twitter accounts! 

***HEADSUP*** – this script may cause you to be throttled by Twitter.  I have added some checking in the Http layer that will let you know if you hit their circuit breaker.   If you hit the circuit breaker, just re-try in 24 hours.

Here is a list of every twitter account for every current Microsoft MVP.  They fit into the following new MVP categories:

  • Business Solutions
  • Cloud and Datacenter Management
  • Data Platform
  • Enterprise Mobility
  • Excel
  • Microsoft Azure
  • Office Development
  • Office Servers and Services
  • OneNote
  • Outlook
  • PowerPoint
  • Project
  • Visual Studio and Development Technologies
  • Windows and Devices for IT
  • Windows Development

The "MVPFollow.zip" file contains a TwitterFollow.ps1 PowerShell script.  Simply run it, enter your "username" and "password" and then tell it what MVP category you want to follow (or type "all") and whalla, you will follow all the Microsoft MVPs with twitter handles in their profiles!  

If you were curious all the cool benefits of being an MVP : http://xavierdecoster.github.io/mvpperks/ 

*update* – the script now supports creating twitter lists (private and public).  It will name the list as "mvp-*" and add the target MVP twitter user into that list.   Also added in some more elegant error handling and follow checking.  

Enjoy!
Chris