Posted on by givenscj

The eBay and PayPal Split

*Posting this now, as our part of the split is pretty much done even though we still have a few more weeks before the final split of the two companies*

As many of you know, I was asked to return as the architect to help split eBay and PayPal into two separate companies.  It was the first time I had ever been involved in "splitting" a public company into two parts.  I have been involved in merging companies, but splitting was a whole new ball game.  eBay had acquired Deloitte to advise on the regulatory rules around the split and thus-ly, they naturally would oversee all of our data split activities and approve my split plans.  Before the serious stuff…how bout some pictures of the team?

Pictures:

eBay Team (Vijiya, Surjan, Venu, Sukanya, Lavanya, Harika, Shanti, Asijit and some hidden faces) – I did not take this pic btw!

 

SPS Sillion Valley and Meeting the "Gordon" himself of "Gordon Biersh" (happened whilst at eBay):

 

Teaching the team how to play flip cup!

 

Of course I owned pretty much everything, so "Tell Chris Everything" was burned into everyone's psych…

 

And now the good stuff that will help the tech world….

Active Directory

  • Users, DLs – Another team built this strategy out.  A very talented team from COO (aka Michael Noel's company).  They used custom built scripts and the expensive yet super shitty Quest Tool (now Dell) to migrate users and then keep the password in Sync.
  • Names that change and those that don't  – DL names did not get updated from "ebay" to "paypal".  As you can imagine, that would have caused a massive workload on all Applications involved.  There was however a filtering process for accounts and DLs that did not need to come over.
  • Email addresses – PayPal went to Office 365/Exchange online.  Until the very last second, eBay maintained the MX records for the "Paypal" domain.  Once the user migration started to occur (laptops to the new paypal domain), the user's email address in the ebay system would be removed.  Of course you just can't flip a switch and automatically everyone is migrated and moved.  It took about a week for single users to really start seeing the migration of the MX and Mail box to occur.
    • Tackling migrated user emails – As you can imagine, workflows and other things that relied on the User Info table for email would show the old "ebay.com" alias, but of course, many users moved to "PayPal".  I decided that rather than leak any future data between the companies via workflows or alerts (and keep with SEC and other rules), I updated all users to "paypal.com".  So users that stay at ebay will not receive an email from the PayPal system.
  • The infamous lost of certain attributes – at some point, someone did something not so bright and annihilated two of the most important AD attributes for roughly half the migrated users.  This of course caused havoc throughout the system once the changes were replicated (SharePoint solutions and 3rd party apps use these properties once replicated via FIM to the user profile).  I was on top of it and made sure the AD guys heard from the SP team before anyone else.  It took 2 days to re-replicate all the properties for 25K users, ouch.

Permissions:

Permissions were a pain in the ass.  The AD guys setup a trust so anyone could login from either domain.  Obviously this is a bad thing cuz then you start seeing the UserInfo table fill up with odd ball stuff.  We had some test trust splits, but I made it a point to cut the ties completely after iteration #3.  I did this by doing the following:

  • Deny the old domain – Add the web app policy that denies any old domain users from logging into the new domain and then update the access denied page to remind them to use their new domain credentials
  • Set the People Picker OU – People would see two of each other when assigned permissions or testing things, of course I would get a defect assigned from every person that thought the migration didn't happen until I got fed up and locked it down to only pull from the local site collection user info list and the new domain.

3rd party apps

  • Vast amount of apps that must split like a cell.  Just to give you an idea:
    • Workday
    • SAP
    • SharePoint
    • Active Directory
    • Exchange
    • Human Resource systems
    • IT Ticketing (Remedy, home built)
    • Video solutions
    • File shares
    • SSO (Ping Identity)
  • Every one of these had to have a project team with PMO, architects, developers and business resources to split them off.  Super expensive and a massive undertaking!

Content – What goes, what stays?
The SEC and other regulatory bodies has some pretty strict rules around what could go and stay with each company.  There were several options one could have used to decide what content stayed and went:

  • Let the users tell us #1 (no guidance) – This requires user's to know what sites they use and what content is important.  Ha…I laughed my ass off at this option.  User's can never find or remember where and what they use everyday.  This would be a failure rigth off the bat
  • Let the users tell us #2 (with guidance) – build a tool that shows user's the sites they have viewed, edited whatever in the last 6 months/year. I built it, but they did not come (or rather, we didn't promote it).  User's never would have taken the time, and the business killed it as we didn't want liability for some reason.
  • Tell them what is going – We chose this option.  We did the migration about four times.  Each time we added more sites to the list of "should go".  At some point, no one complained about data missing.  See the next set of data points…

 What we used…

  • My algo ruled – We ended up using an Algorithm I built to decide what should go and what should stay.  It went something like this:  If the site used the "PayPal" master page, its good.  If the site had an ownercreater with a "Paypal.com" email address, its good.  If the site title had the name "PayPal", its good.  If the site didn't meet the first set of criteria, but had a document with the doc name having "PayPal", is was called "Mixed", it also got to go.  Legal loved the algo, signed off, and away we went.
  • Have a process – As much as the algo proved its value, there were still a set of sites that had to go no matter what.  The main intranet site was not tagged as "PayPal", but still had to go.  Shared resource sites managed by "eBay", had to go.  We put these in a special category and user's had to tell us they wanted them or they would get deleted.

IRM – Information Rights Management

  • IRM Enabled Docs – Turns out when you migrate from one domain to another, you lose access to all IRM enabled documents.  Huh…who would have thunk it?

Branding and MasterPages

  • 2007 strikes back! – 2007 had some weird crap in it.  My domain knowledge from the last upgrade helped me to "fix" some SP Hive files to support weird things like missing web part zones and other oddities
  • Lucky lucky – Our site creation process from 3 years ago helped us so much.  All the sites would get tagged based on the user and the templates they selected.  It made it super easy to find the PayPal sites (based on the algo above).
  • PayPal Branding – All migrated sites got PayPal branding, period.  No other companies, no eBay, therefore all master pages and other images were moved to PayPal.  It caused a bit of grief for some sites, but minor.

Single Sign On

  • SSO to SSO – all those apps above…yep…SSOclaims enabled.  eBay has a mandate that *all* new applications much support claims based Auth.  If not, you won't sell it or install it!  This meant migrating every new PayPal instance to the new PayPal SSO.  Not an easy or quick process of course.  I have another post about using the old Kerberos shite coming soon.

URL Migration

  • SystemUpdate – when updating listitems and files, make sure you use SystemUpdate.  It just looks tacky having a bunch of updates for URLs as people find others that have to be updated.
  • Keywords  and Best Bets – ensure that you remove old best bets (ones that point to sites that were deleted) and also make sure that you update the best bet urls so that they point to the new urls
  • InfoPath forms – forms that submit data to lists or other libraries must be updated.  I have a post from the last migration that details the hidden awesome code for in-memory InfoPath updating here
  • Title column and 255 chars – some URLs will get bigger when you replace them. Which means if the users decided to use the damn "Title" column, you will know that you can't change its type from Single Line of Text to Multi-line.  Those items are a lost cause and must be written off as much.  For the other columns, I could easily change the type to multiple lines of text and then update the URL.
  • Crawl Rules  – be sure you update your crawl rules to exclude those really dumb emails from customers that have weird stuff in them that should not show up in search results.  And all the unaccetable keywords that are not allowed in ebay and paypal systems.
  • Profile Properties – make sure you update User Profile properties (such as PictureUrl) that point to the old system to point to the new system
  • Code and Layouts files – Yeah, if you code has static AD domain references, you will have to update that.  And make sure that weird statically coded URLs in layout files are also updated.  This is super bad practice to statically code your absolute urls in your freakin layouts file.

List template and User Solutions

  • List Templates – Just blow that shite away…its ridiculous to try to create something and see 100s of weird list templates showing up
  • User Solutions – I blew all these away, totally worthless

Code, maintaining two code bases

  • TFS – yeah, you have to create new branches to support the changes in each company.  And at some point, they must break the link completely.  It required some serious refactoring in some cases to support both companies with one codebase.  I learned a ton of lessons around when you should put a company name in and when you should not.  In a majority of cases, NEVER put the name of the company somewhere in code.  Horrible, just horrible.
  • Company Names – and speaking about company names.  Never create columns on your lists that have the company name, especially if they will be used as refiners in your search center!  PayPal will forever more have the name "eBay" in the search results.  Bummer…

Testing, testing, testing

  • Have a tool that doesn't suck – When browsing, editing and doing reporting, don't use HP ALM.  It is a horrible piece of junk.  That being said, find one that at least:
    • Allows usersdepartments to enter defects
    • Allows you to easily find defects (assigned to you or the project)
    • Ensure user's don't add to a defect rather than creating new defects (this drove me crazy and I went off on a couple of people).
    • Set priorities for resolution
    • Set Severity and resolution times
    • Make sure people know that a defect will roll to the next migration attempt (deferrable)
  • Have a killer defect manager – we had an awesome lady running our defect management, Sukanya Samal.  Very knowledgeable and very to task, just amazing.  You never know when you might have some jokers like these fellows…
  • Have some great tools – I created several tools three years ago that help with checking the health of the migrated farm.  You should too:
    • Link Checker – checks all the links across the farm recording what pages have what links.  This allows you to go back in and query the pages that didn't get all the links updated.  You can then dynamically run your Url Updater to target just these pages after you have made modifications for the outlier cases
    • Error Check – a tool I have had for a while that checks all SharePoint pages for error conditions (correlation errors, web part error, asp.net error).  Every time this would tell me if someone forgot to install a 3rd party product or some other dependent assembly somewhere.  I would then be able to fix all the pages.  It was also useful for determining pages that were also broken in production that we didn't need to fix in the migration (don't get blamed for someone elses problem!)

Scripts, scripts and more scripts

  • Script everything! – I had over 85+ scripts for migrating the environment.  We could do a full rebuild in about 3 days.  Examples:
    • Install FarmAdd servers to farm
    • Install solutions
    • Create app pools, web apps and service apps
    • Start WFE and App services
    • Add web config entries
    • Attach CDBs
    • Create your web app policies – to add all your testers and admins
    • Create AD and BDC connections, map UPS properties
    • Deploy solutions
    • Update workflows
    • Migrate Users and Groups
    • Create Content Sources
    • Setup Best Bets (Fast and SP)
    • Delete Sites
    • Find and Replace URLs
    • Enable Full Auditing across sites
    • Setup Trusts with certs
    • Set and unghost master pages
    • etc, etc….
  • Utilize script references – I had several scripts that held our common functions and variables.  This allowed us to modify one file and have all the other scripts target our environments (Staging, QA, dev, production) with no changes.
  • Order Matters – don't just fire off scripts at random.  They must follow a logical order.  Take for instance…update the workflows to an existing user, then fire the MigrateUsers command.
  • Table locks matter – Same reason your OneDrive doesn't sync more than 5000 items, watch for when your scripts query over 5000 items in the DB, this causes a table lock and will stop all other scripts.

Of course, my scripts became famous:

 Migrating Users and Groups

  • Migrate cmds –  If you have ever done a domain to domain migration, you will know the pain involved.  How do you find all the users to call the migration usergroup command on?  How long does it take, and how should I execute it?  Well, turns out that you have to query *every* content database and the UserInfo table to get *every* user and group out.  You could then feed that to a PowerShell script that would update them.  But running that script as a single thread will take DAYS!  Especially with 60K+ users.  You need to create a dynamic script that queries the UserInfo table and then takes an input *char* such as "a".  You can then fire off 26 individual PowerShell scripts that cut the time rough by 26.  This runs so much faster.  Oh, and that same pattern, you do for EVERYTHING involving users.  Make a script that takes a "char" parameter and get it done oh so much faster!

Firewalls and proxy filters 
PayPal went security crazy.  They locked down everything.  Waaayyyy more than eBay.  You couldn't do anything without getting flagged for it in some way or another.  And just about every subnet was isolated from every other subnet.  Ouch.  Some tips…

  • F5 VIPs – ensure that your servers can use the same VIP for load balancing from inside the subnet!  This drove me crazy until I figured out what was going on.  Ensure the F5 guys add the right iRules to the VIP!
  • Outbound proxy filter – damn, when you block your own web properties and wonder why things don't work?  I find it funny that a company will lock down the outbound internet traffic to their own sites, didn't make for a very workable "crawl" system for those sites! 
  • Teams that just can't do it by themselves – eBay is going to Node.JS and Druple with ElasticSearch.  PayPal is going *ALL* Microsoft.  The new intranet team that run the eBay side were not network guys.  And they kept driving me crazy because they couldn't connect to SharePoint and saying it was my responsibility to fix it.  Bullshit.  They just needed to quit being lazy and go talk to the network guys.  Just stupid that "architects" don't know how to diagnose and resolve network issues.
  • FIPS policy – they tried to apply it, but I caught them when they tried it.  SharePoint don't work with FIPS.  It's not that SharePoint isn't secure, its just that it used some non-FIPS algos for hashing some things for speed.  Of course, enabling FIBS causes .NET to exception out anytime a piece of code uses it.  They dropped that pretty fast when I told them how bad it would be for SharePoint and all the other .NET apps on the network.  Reference this

Networks:

  • IPs, subnets and VIPs – oh my!  Make sure your subnets are setup property…reference this

Workflows:

  • The infamous workflow error I surfaced years ago (on top of many others) – This kept bitting me in the ass.  Every migration, every time, without fail.  Some workflows would update fine, some, not at all.  Some would update, but one file would remain.  Retarded.  I eventually had to manually fix association and workflow instances.  Not for the faint of heart.  Reference this post.
  • MigrateUsers and workflow associations – And just if injury and insult wasn't enough…death ensued.  If you run the MigrateUser command and the workflow has an invalid user on and file.Properties["vti_modified" property, you will lose those associations.  Reference ths post.

Audiences:

  • We created the UPS from scratch.  Which means all things related to it were re-created, including audiences.  This meant that all audiences got new IDs.  These new IDs did not map to the audience settings across the list items and pages of the farm.  Ouch.  We had to update all those (but with a script of course).

Notable Twitter Handles:

Summary:

Splitting a company cost a lot of money, time and resources.  I feel sorry for all the people that will have to find something to do (if their sales team doesn't close some deals and find them work) after we are completely done. We are talking 1000s of people to split eBay and PayPal.  They gotta go somewhere, the world keeps spinning and other companies have cash to burn so I guess they'll take some of this knowledge and find another project.  I know that I increased my intrinsic value from this project.  If anything, I now have a new monkier…"Chris Baba".  And some followers:

It seems to be incredibly rare that you get asked to come back and do a project at a company almost fours years later.  Especially of the size and important of this.  Just as it was 4 years ago, this has been an incredibly exciting and amazing experience that I'll cherish as a major accomplishment to add to the many I have worked so hard for.

Hope you learned something to support the migrations you are doing or are thinking about doing, especially those that are about splitting a company off from another!

CJG aka "Chris Baba"
@givenscj

Post Views: 1,312