UPDATED (1/23/2014 4pm) – Added info about the Web Proxy features and quick mention of using Azure Message Queues
What happened to the days of simply having usernames and passwords? Those were the good ole days for developers…simply a "test" user, with the password "test" was all you needed to do anything! Now a day's you can't get by thinking like that. However, it was so damn easy…but as easy as it was, it didn't let you do things like "Test as a target user" without having that user's password. So here we are, the days of OAuth 2.0 upon us and although many have conquered it as brilliantly as the sun shines in the sky, those lovely people are a very select few. So after two months of browsing and answering questions on the MSDN forums, and some chats with the SharePoint engineering team, it dawned on me that no one has really done a simple straightforward, no-BS, post on all the ways to do auth in SharePoint and Office on-prem and in the O365 cloud when making a simple REST API call. There have been some interesting high-level posts such as this – http://msdn.microsoft.com/en-us/library/fp179887.aspx, but the basic user will gloss over after the first PGDOWN. Therefore, this is my first go at it…
Some preamble…I'm not going talk about how Windows does what it does, or how to configure Kerberos, or how to configure an STS. Nor will I talk about how FedAuth cookies and Bearer tokens are converted to SPUser objects and claims identities, nor do I discuss assigning permissions to these entities (http://msdn.microsoft.com/en-us/library/jj687470.aspx). Honestly, that stuff is really the simple aspects of the equation and has been knocked out of the park by many other individuals (beaten to death?). What this blog DOES focus on is the final result of these mechanisms such that you can make a simple HTTP REST call!
Why this post? Other than the "drop the simplicity hammer" aspect, it is to show you can move from the older sandboxed solution compiled coding style and the server based code deployment models that is now frowned upon to the new REST model without the App Model included in the conversation!
No OAuth – THE OLD SCHOOL WAY
If you don't care for the "App Model", then don't use it. But you will be in the world that I introduced above…simply running as the account the application logs in as. In a lot of cases, this is totally fine. I have been swimmingly moving along with writing tools and other things using this approach. As it turns out, the current App Model's auth approach just isn't ideal for many of us (more about that later).
In terms of non-App ModelOAuth, here are the possibilities:
- On-premise with Windows username and password
- This is one of the easier methods to do auth (given that all your apps are running on a windows platform anyway). What happens here is that when you do the request, your "application"'s HTTP request will be challenged to provide credentials. If you are using a browser, the user is prompted for the credentials and then those are passed. If it is your own custom application, you must pass those credentials via the "Authorization" header in the HTTP request via the "Credentials" of the HttpRequest object. If the request is successful, you will be provided a FedAuth cookie value. From there on out, you simply pass this FedAuth cookie in all your requests and not the Authorization header. Simple, easy…LOVE IT.
- Example:
Util.DoGet(url + "/_windows/default.aspx?ReturnUrl=%2f", "");
//inside the DoGet:
if ( username.Length > 0 && password.Length > 0 && domain.Length > 0)
req.Credentials = new NetworkCredential(username, password, domain);
//parse out the cookie
fedAuth = ParseValue(cookies, "FedAuth=", ";");
- On-premise with Forms-based username and password (membership provider)
- This is similar to the Windows based version, but rather than be challenged, you go through a series of redirects of pages in the SharePoint site. Forms based auth requires you to setup the provider in central administration. Once setup, you will be presented a page to enter your credentials, from here you post the values back to the page and the ultimate result is yet again, a FedAuth cookie that you will pass with all your requests. This type of auth is not supported in O365
- On-premise with Forms-based username and password (STS method)
- Similar to forms-based auth but rather than it be a local membership provider you had to install,
the auth page will redirect you to an external party's site where you will login, and then some data (a token) is returned to sharepoint. The result? Yeah…a FedAuth
cookie that you will pass with all your requests from there on out. This one requires you to setup a trust between your STS and SharePoint, this is almost identical to the App Model method called "High-Trust". The actual details of what is happening behind the scenes is not important, but you will need to know what the format will be of the POST you have to make to the remote STS login page (this will be different for everyone).
- O365 Authentication (MSOnline username and password)
- O365 doesn't support Windows based Auth per-se (see next option). As you know, when you go to a O365 tenant, you get redirected to the https://login.microsoftonline.com auth page. From here you will type your tenant username and password, and then be redirected back to the O365 tenant. The result? Yeah…a FedAuth cookie, but you also get an extra one, the rtFA cookie. Both of these will need to be passed in your subsequent requests.
- Example:
//start on auth page
string authUrl = url + "/_layouts/15/Authenticate.aspx";
Util.allowRedirects = false;
Util.DoGet(authUrl, "");
Uri uri = new Uri(url);
//redirect to the forms page
Util.DoGet(uri.Scheme + "://" + uri.Host + "/" + location, "");
//redirect to the O365 login
string html = Util.DoGet(location, "");
string mspOK = ParseValue(cookies, "MSPOK=", ";");
cc = new CookieContainer();
cc.Add(new Cookie("MSPOK", mspOK, "/", "login.microsoftonline.com"));
string ppft = html.Remove(0, html.IndexOf("PPFT"));
ppft = ppft.Remove(0, ppft.IndexOf("value") + 7);
ppft = ppft.Substring(0, ppft.IndexOf("""));
string postUrl = html.Remove(0, html.IndexOf("post.srf?") + 9);
postUrl = postUrl.Substring(0, postUrl.IndexOf("""));
url = "https://login.microsoftonline.com/ppsecure/post.srf?" + postUrl;
string post = "login=" + username + "&passwd=" + password + "&PPSX=PassportR&PPFT=" + ppft + "&n1=111950&n2=-1389325702000&n3=-1389325702000&n4=111952&n5=111952&n6=111952&n7=111952&n8=NaN&n9=111952&n10=112024&n11=112021&n12=112270&n13=112272&n14=113180&n15=41&n16=113269&n17=113269&n18=113276&n19=379.53448033278346&n20=1&n21=0&n22=0&n23=1&n24=35.764367179524925&n25=0&n26=0&n27=0&n28=0&n29=-1389325815165&n30=-1389325815165&n31=false&n32=false&type=11&LoginOptions=3&NewUser=1&idsbho=1&PwdPad=&sso=&vv=&uiver=1&i12=1&i13=Firefox&i14=26.0&i15=1920&i16=952";
string response = Util.DoWorkPost(url, post, "");
string t = response.Remove(0, response.IndexOf("id="t""));
t = t.Remove(0, t.IndexOf("value") + 7);
t = t.Substring(0, t.IndexOf("""));
string action = ParseValue(response, "action="", """);
post = "t=" + t;
DoWorkPost(action, post, "");
rtFa = ParseValue(cookies, "rtFa=", ";");
fedAuth = ParseValue(cookies, "FedAuth=", ";");
- O365 Authentication (STS)
- As mentioned previously, O365
doesn't technically support Windows based Auth. But it can be made to redirect to a STS (like ADFS) such that the user signs in using a federated set of credentials and then redirected back to SharePoint Online. Basically this is the same as if you did all the work on-premise (STS). The result? Yeah, you guessed it…a FedAuth cookie and the rtFA cookie. Both of these will need to be
passed in your subsequent REST requests.
Oh…did I mention that all of this is HTTP based? Anything that is HTTP based can be scripted on any platform, any where, any time, as is all of the above. This is by far, the easiest and most efficient way of creating applications that integrate with SharePoint, Office and Project server. Did I mention…this was simple?
OAuth and the App Model – THE NEW SCHOOL WAY
Apps, apps and more apps. So as easy as the above methods are, what is the issue? Context. When you login using the methods above, you are locked into running everything as the logged on user (that you have the password for). That can be very limiting when you want to update an item on-behalf of someone else yet have it show as if it was done by that person (and deploy that app to 100's of thousands of people around the world). Hence, OAuth was created to do just that. The ability to run as a user with all the permissions (or a subset based on what they allow you) in your application is pretty awesome. But…it's kinda scary right? Don't be scared…every person that has a Facebook, Twitter, Linked-In or any other social media account has already done this in one way or another! So let's take a look at how auth works in the various models when it comes to SharePointOffice Apps:
- SharePoint hosted app – on-premise
- First of all, doing everything in JavaScript exposes your hard work. This means people can take it, change it, steal it, and republish it (yes, this has happened). As someone that is focused on protected IP (and securing the revenue that it will generate), I'm not sold on this and likely never will be. JavaScript and Typescript and whatever name you think of later (angular, knockout, blah) is simply a means to make a pretty looking and responsive UI. That's it. Not business logic that you want to protect. SharePoint hosted apps will only allow you to deploy JavaScript, no custom code hidden by compilation. That being said, if you don't have anything important that you'd like to protect and its simply a fun and helpful app with no dollars tied to it, then create a SharePoint hosted app. All of the auth will be taken care of for you if you use the .NET CSOM or the cross-domain script library (more on that later). Oh, and did I mention that you don't need to expose your on-premise SharePoint to the internet (unless you implement your own message queuing to Azure)? This is a very easy and simple approach. Oh and whilst were at it, be sure you don't get on my bad side if you do build one and publish it (https://blogs.architectingconnectedsystems.com/blogs/cjg/archive/2013/10/05/CJG-Review-of-All-SharePoint-Store-Apps-and-SPStore-Automater_2100_.aspx). UPDATE: I did miss one important aspect of using SharePoint hosted Apps as "shells". You can obfuscate your code by using the Web Proxy behinds the scenes to call your provider hosted endpoints (see http://msdn.microsoft.com/en-us/library/office/fp179895.aspx?ppud=4). There is a serious limitation with this, you are limited to 200KB in the response!
- Provider hosted app (High-trust) – http://msdn.microsoft.com/en-us/library/office/fp179901.aspx
- In this model, you create a SharePoint app, deploy it and attempt to click on it. The problem most people will run into is the fact that you need a certificate on the provider side that will encrypt the user's window token and pass it back to SharePoint. SharePoint then needs to know it can trust whatever is coming across the wire so you need to do some PowerShell magic to enable this, eck…"setup work". Since you can't install certificates in O365 like that which is needed with this, you can't do these types of Apps in O365. The result of successful wire up of the trust and certificates? A Bearer token. This token can be parsed from the TokenManager code (or your own parser) and then used in your REST API calls from JQuery. Some of you may be asking…Why? Simple…using my SP REST tool (http://sprest.architectingconnectedsystems.com/) you will notice that you won't be able to do everything you want to do and may need to call the CSOM XML endpoint directly (and since some of you will build a non.net based application, .NET CSOM is not an option, but the JSOM is, but eck goodbye IP). In this case, your app is hosted internally on your network and as such, is not meant for worldly consumption.
- Provider hosted app (Low-trust on-premise) – http://msdn.microsoft.com/en-us/library/office/dn155905.aspx
- In this model,
you create a SharePoint app, deploy it and attempt to click on it. You will get an error that ACS is missing. You have to actually enable ACS access (which means you need an ACS account to store your data). This also means that your environment will need to have internet access to ACS in order for everything to work properly. Again, ultimate result of your work…a Bearer token.
- Auto-hosted app (Azure) (aka: Provider hosted app (low-trust O365)) –
- In this model,
the provider web is deployed to Windows Azure (auto-magically) and the App must be
deployed to an O365 SharePoint tenant or ACS enabled on-premise installation. No exceptions. The O365 platform has a special Azure Control Service Application that will handle
the auth part (which you learned is not enabled by default in a on-premise install). Again the result is a Bearer token that will be passed in
all the requests but you don't get to see any of this via http requests and responses since you can't run fiddler between the two. However, you can see it in your .NET code.
- The Missing Link (provider hosted, forget trust, it's OAuth!)
- So what is missing here? Why do people get confused? Here's the
answer. The naming conventions are less than ideal. High trust? Low trust? Does that seriously tell you anything? Trust of what? The permutations of configuration are insane (just look at the options above). Those of us that have been around awhile (myself included),
remember when OAuth did what it was supposed to do. Simply have a
clientId and clientSecret. No "high-trust" weirdness, no "I need Azure
Access Control Services", etc. When a user clicks on an app from the platform, it will
simply send a time-limited access token to the remote application, that
application includes that token in all future requests and life is
good! So what happened to this option? Honestly…I don't
know…can't tell you. They made it incredibly difficult for those of
us that want to protect our IP, yet sell it to the masses to make your
lives easier. Big gap right? Why are there so many configuration steps needed in order to get a provider hosted app to work? Customer's hate this, ISV's hate this…I hate this. We need an option that simply works without any crazy customer setup steps!
As a separate side note: The problem with provider hosted apps that live on your servers, yet you want others to utilize, is that the target SharePoint farm must be accessible via the internet. As you can imagine, that is a NOGO for many people. But those of us that want to build the really cool exciting apps and make money off them, don't want to build a JavaScript based SharePoint hosted app and lose our IP. This is the catch-22 that exists right now. SharePoint Online has been designed with security and internet access in mind. That makes it a perfect fit for Azure and provider hosted apps. Problem is that it has not fully replaced (maybe will never?) on-prem installs. This points to provider hosted apps currently fitting the O365 model more than anything else when it comes to no-setup and ease of install. But if you go the whole route of web based UI, you have to deal with cross domain scripting issues…cross domain scripting wha???? Yeah, poke me in both eyes with a hot iron…
Cross Domain scripting…blah blah…oh whatever!
Creating your own windows and console apps that simply do posts to endpoints with the bearer or fedauth tokens = EASY. Creating web applications that have heavy use of JavaScript and cross domain requests = HARD. Once you get into the cross-domain library (and this one) (a hack to send your request to SharePoint so it does the work for you behinds the scenes bypassing the browser security) the code gets an extra layer of complication and starts to look really ugly (hard to debug much?). Why put yourself through all that pain and anguish? I just don't get this, it seems like a problem was created a long time ago, a patch was layered on top (which complicated things further), then someone wrote this, then that…and here we are. A tangled mess of web crap all over the place. "Why so serious?" Don't make things more complicated than they have to be people! I guarantee I'll build an app faster using the old approach, demo it and sell the idea before you can figure out if your clientID is right or not.
Forcing the Issue
*akward silence*….soooo….here we are. Me…being me. Asked me. Can me figure out a way to bypass this oddness? Me did. Can I register an application simply with a client id and client secret (via an app package) and then make calls to SharePoint (no funny configuration business), no apps, no cross domain stink? Yes, you can. But like the ways of old, you will be stuck running as that App and only that App. An endpoint that people don't talk about (because people don't go this deep) is the "http://localhost:32843/SecurityTokenServiceApplication/appsts.svc" endpoint. This endpoint is responsible for generating the tokens! Once you have deployed an app package to your farm (App manifest must set the AppOnly flag see http://msdn.microsoft.com/en-us/library/office/fp179892.aspx), you can make direct requests to this endpoint to get a context token as shown in the next set of code from any application on your network!
PowerShell Code:
$appId = "<REPLACEWITHCLIENTID>"
$spUrl = "http://teams.contoso.com"
$spWeb = Get-SPWeb $spUrl
$realm = Get-SPAuthenticationRealm –ServiceContext $spWeb.Site
$fullAppId = $appId + '@' + $realm
$appPrincipal = Register-SPAppPrincipal -NameIdentifier $fullAppId -Site $spweb -DisplayName "OAuthApp"
Set-SPAppPrincipalPermission -appPrincipal $appPrincipal -site $spweb -scope "Site" -right "FulLControl"
C# code (using sts service reference):
SPAppSTS.rst rst = new SPAppSTS.rst();
rst.nameId = "<FULLAPPID_FROM_ABOVE>";
rst.requestType = "application/issue";
rst.appliesTo = "00000003-0000-0ff1-ce00-000000000000/teams.contoso.com@<REALM_FROM_ABOVE>";
SPAppSTS.ApplicationSecurityTokenServiceContractClient client2 = new SPAppSTS.ApplicationSecurityTokenServiceContractClient();
SPAppSTS.rstr rstr = client2.Issue(rst);
Console.WriteLine(rstr.rawToken);
Console.ReadLine();
You can parse out the bearer token from the rawToken and use it in all subsequent requests! No need for any crazy ACS setup or certificates! Boom…. What is the drawback here? You can't hit this endpoint in O365. So it is an on-prem tactic only. Where did the documentation for this approach get missed? LOL….again…who knows.
Back to the Basics – FedAuth and Bearer tokens
Ok, back to the goal of this post.
As you can see, regardless of the path you take (or the battle that you have to fight to get your token, or the battle to get your request to actually get sent to SharePoint across domain boundaries), it all boils down to the FedAuth and Bearer tokens. So, what happens if you "lose" the FedAuth or Bearer token? I'll simply say…DON'T. And it is a bad idea to generate these cookies in one process and then pass them to other application processes. Can you do it? Sure! It's HTTP, stateless. It has no notion of the before and after. As long as the request has a valid FedAuth or Bearer token, you are golden! This means if someone picks it up in the traffic that is sent across the wire (oh you don't use SSL on all your web apps? Ask Spence about that, you'll get your yearly quota of bad English words spewn at you), they can impersonate the userapp. I personally would love to see some kind of security levers than can be tied to these very important objects such that IPs or the User-Agent must match. In the presence of these levers, if these values don't match…well…that means you simply need to login again. (Spec writing time?)
Helping you troubleshoot your own Apps or other people's Apps
So what is the end result here? Make sure that your HTTP requests to SharePoint include the proper cookies to allow your request to be executed. If you are doing an application on-premise, make sure that the FedAuth cookie is present in all your requests. If it is an application that is hitting O365, make sure that the FedAuth and rtFA cookie is present. If you are using Apps to generate an auth context, make sure that the Bearer token is passed in the requests. It IS that simple (although the methods to get here con volute this fact). Just using this knowledge will help you with troubling shooting things like Business Intelligence with Excel, and a whole myriad of other things!
Future
So where are we headed? How about a
single FedAuthBearer token to rule them all? This token can be passed
to anything, anywhere, anytime and the context that is provided allows
you to make calls to Yammer, Exchange, Lync, SharePoint, Office Web
Apps, etc. I have no doubt that something *like* this will show up in
the future. It only makes sense to do so. Imagine the rich
applications you can build if you were allowed to modify Exchange via an
app based on a user profile setting in SharePoint? Or an app that
sends a Lync message when someone on Yammer mentions you? This is the holy grail of building a cohesive platform that has any real meaning for Microsoft (hmm, follow Google's lead?).
I can tell you for certain that the App Model as it is currently designed will not be the same one that exists (or is even compatible) with its next incarnation. I can't tell you that using the easy methods above can will always work as changes in the auth flow or cookie names can always happen. In the end, you can be guaranteed that where we are today, will not be where we are in the future, no matter what simple or complex paths you chose from the options above.
Summary
If you can get away with building an app that doesn't use Apps and will meet the requirements with a generic user….DO IT. The time you will waste deciphering all the broken ,half working examples and misguided posts spread across MSDN will equal the time it would take you to build it using the simple methods above and…you'll have time to have *several* Stone Brewery (or your favorite brewery) brewski's with your friends and family.
