Dynamics CRM Upgrade: 2011 to 2016, Part 3

A quick recap of parts 1 and 2 of this series, Dynamics CRM Upgrade: 2011 to 2016:  In part 1, we looked at some general considerations with a CRM 2011 to 2016 On Premise migration, such as resource planning, assessing new features, test and release planning, and the final roll out. I introduced my current customer experience with an enterprise system migration of a very complex, xRM heavy CRM 2011 on premise system to CRM 2016 on premise. In my latest post with part 2, we focused on the database portion of the migration, such as the staged approach and moving the upgrade through different iterations. We also discussed the impact of these requirements on resources, timing, and overall planning.

In this post, we can take a look at some of the issues faced when backing up, restoring, and importing our CRM 2011 database.  We can also take a look at a few things that might help streamline the process.

Some bumps in the road

The staged approach entails importing our organization database through multiple versions of CRM in order to complete the full database migration. As mentioned in part 2, once the organization import is complete, we should have a working CRM 2013 system.  As discussed, my customer has a heavily customized xRM system, which means many plugins have been registered and deployed to the database. With the import and upgrade, these plugins are still registered and SDK message processing steps are ready to fire. Unfortunately, a completely new CRM version means that our old plugin code will not work! Because this is a major version release, our CRM 2011 plugin assemblies will fail to load causing exceptions. If we do not trigger an event for which a plugin is registered, such as an on Create plugin, this should really not pose an issue. But my customer registered a Retrieve Multiple plugin that fires each time we brought up a list of records. This meant that we could not work with most of the system, either viewing our customer data or accessing the administrative screens.

In order to avoid this, we simply disabled the Retrieve Multiple plugin before performing the Organization import on CRM 2013. In our case, we did not want to impact the production system so we did not disable the plugin step on CRM 2011.  Once our database was restored to the CRM 2013 SQL Server, we run a SQL script to disable the plugin execution steps BEFORE we ran the organization import.  With this SDK message processing step was disabled, we could navigate our system and only ran into issues if we triggered the other plugins, such as an On Create or On Update plugin.

Another issue we faced early on occurred the SECOND time we ran the import. We pulled a new backup from production because my customer made some code and configuration updates and it was time to merge our updates. I had hoped to preserve my existing tenant on my development virtual machine in case we had issues, so I restored the new backup as a new database on SQL server. When attempting to import the organization using the Deployment Manager Organization Import Wizard, I received an error indicating that the Organization GUID already existed. Because I was importing a backup from the same source system, the Organization GUID clashed with the existing Organization registered in the configuration database. I have seen a few articles on how to change the Organization GUID so that you can avoid this clash. I don’t want to post any examples here, since I have personally not attempted any of the methods I’ve found and I really do not believe these are supported! Also, because my customer wants to replicate the database import steps as close as possible to the final go-live migration, we take the step of deleting the existing database and performing a new restore each time we pull in a new Production backup. My customer’s infrastructure team also provide some additional development environments that made our work much easier!

Another odd issue that we faced occurred during the CRM 2015 database import.  We had successfully imported the database into our CRM 2013 instance and after the backup and restore to our CRM 2015 instance, we kicked off the Deployment Manager Import wizard. The import failed and after some digging into the log files provided by the import wizard, we saw that one of the Activity Feed plugins attempted to fire. It seems that these were updated during the CRM 2013 import and they now caused issues with the CRM 2015 upgrade process. What was also confusing is that the Activity Feed plugin SDK message processing steps are not displayed by default within the Default Solution. This was a relatively easy fix once we realized what was going on.  The steps we took:

• Navigate to Settings -> Customizations and choose Customize the System
• View SDK Message Processing Steps and select View:All to see the Activity Feed plug-in
• Select All SDK Message Processing Steps and Deactivate

Once we performed these steps, we backed up the database again and the import on CRM 2015 worked great.

Streamline the process

During the development period of upgrading the system, my customer had to run the migration several times.  In order to speed things up a bit during the migration, we took a few steps that seemed to save us some time.  As mentioned in my earlier posts, even just copying the files between servers can take some time!  So a few of these approaches might help.

• Disable plugins and workflows – I had assumed that when performing an Organization import, only the database would be affected and the rest of the system was essentially offline.  But after running into the issue with the Activity Feeds plugin mentioned earlier, I am assuming that this is not the case!  So if you have the option of disabling these during your import steps, I am thinking that it would reduce some of the time to complete. This is one of those items for which I would like to get some clarification.  If anyone knows in the meantime, please drop me a line, and I if I get an answer, I’ll update the post.
• Cleanup your DB! – I am sure most reading this have some experience with your CRM databases increasing in size over time. One very popular culprit is the AsyncOperationBase table.  The following support article provides details on the issue and some scripts will help to clean things up: Performance is slow if the AsyncOperationBase table becomes too large in Microsoft Dynamics CRM. If your organization has run into this issue, you likely have some of these steps already in place to run periodically. Another clean up script that can clean up some space is to clean up Outlook client sync data. This article by Sean McNellis provides a good summary and the script: Cleaning up CRM Sync Entry tables. These are older posts, but remember that we are running against a CRM 2011 database. I run each of these scripts, and a few other specific to my customer’s system, right after I import the database on the CRM 2013 SQL Server. Here is yet another blog post by Ben Hosking about database cleanup and maintenance: CRM 2011 – Cutting my CRM database down to size This is another topic on which feedback and suggestions are welcome.
• Prune some data – The cleanup steps already mentioned will clean up your database, but what if your system has a LOT of old data that is no longer being used?  Does your system have an archive or data warehouse to which you periodically push data? If so, this is an additional step that could shrink your DB and simply give the import process less data on which it needs to work, and moving the backups. This step is completely dependent on your organization’s situation and may simply not be an option because of internal or external regulations, but it is worth investigating early on as it may have an impact on your delivery.  Even if it’s only to force your team to review its data, processes, and long-term data management strategies, it’s a good exercise if you can spare the time!
• Just drop some stuff – Once again it’s about reducing the size of the database, but here, you can drop some data temporarily. My customer deals with a large number of attachments and annotations. They regularly use the Note attachments for moving around reports, sharing documents, etc, in addition to their syncing of email with their system that might contain attachments. So the dev team decided that during the their development cycle, we would simply drop all annotations from the system prior to import. Unfortunately, I don’t have the exact number, but during one migration test, the compressed database backup shrank from 50GB to 9GB.  This really cut down on moving files around as well as the database restore, and each stage of the import from server to server.  For multiple iterations, this really saved us time, but during the actual migration, we will not be dropping the annotations.
• Bump up resources – Throw hardware at it!  If you have the resources available, I would suggest bumping up the CPUs and RAM on the target environments. The database restore and the Deployment Manager import wizard can really use a lot of memory and CPU, so an increase during running these times. In my customer’s case, they have a virtualized environment, so they have some flexibility in managing resources.  And during the actual go live, planning for these additional resources can definitely reduce your downtime. Even if the calendar time remains, say you block off a weekend for the upgrade, giving the migration team a few extra hours to deal with issues or run tests can be an enormous help.
• Time it accordingly – One point was brought up just today by my customer that I had not even considered, but it can really make a difference, so I definitely want to note it here. During our weekly call, she asked about the timing of when I was testing out latest backup because of possible network latency. Their organization has nightly batch processing that can chew up network bandwidth.  Even though the CRM system will be offline during the actual migration, other systems will not. If we decide to perform our migration when the network is slow, it can really eat into our time to complete the upgrade. So this may not be an issue given other networks, but it is definitely a great point to keep in mind when planning!

UPDATE

I was speaking with Sean McNellis, the Premier Field Engineer (PFE) that has been working with my customer and who provided the cleanup script above, and he brought up an excellent suggestion that can save a significant amount of time.  It’s very simple and I feel silly for not considering it earlier.  With each of the steps mentioned above, I indicated that you need to perform a backup, copy, and restore to each subsequent system. Meaning, run the Organization import wizard from each subsequent server, but direct it to a database on the final target server.  So we would backup our current production database and restore it directly to our target production environment.

The steps are very similar to what I had previously outlined part 2 with the exception of no longer needing to perform the backup, copy, and restore for each upgraded version!  I’m going to just steal Sean’s steps directly, listed below:

1. Take prod offline
2. Backup the DB from prod
3. Restore on the future prod SQL Server
4. Import into 2013 – from SQL Server from step 3
5. Disable/drop org from 2013
6. Import into 2015 – from same SQL Server from step 3
7. Disable/drop org from 2015
8. Import into 2016 – from same SQL Server from step 3

In my customer’s situation, since the DB is nearly 150GB, this can save a significant amount of time.  My rough guess based on our dry runs for the migration, it may shave nearly 3 hours off of our downtime.  We can also plan resources accordingly,such as adding loads of memory and CPU horsepower to the target server versus adding resources across the board.

This is why you always have someone review your work!

Up next

Once these migration steps are completed on your database and you have run through gauntlet of database conversions, we get to dive into the rest of the system.  In the next post, we will start looking at the next steps in bringing the system up to speed on CRM 2016. This includes looking at the changes to the database and its impact on our solution. new CRM components such as an updated entity model, and then diving into the code changes for our solution.

As always, comments, suggestions and corrections are always welcome!