External Data Integration via Virtual Entities
Virtual Entities are finally here!
Ok, so the Virtual Entities feature for Dynamics 365 CRM has been available for quite a few months now… but it’s a big deal for solutions requiring external systems integration. The article Interact with data from external systems using the new virtual entities by Jesper Osgaard of Microsoft Denmark provides an excellent overview of Virtual Entity capabilities and considerations when deciding whether to leverage the new feature. Also read the follow up post about setting up relationships: Virtual Entities – Relationships
In a nutshell, Dynamics 365 CRM now offers the ability to configure an entity whose read only data lives outside of the CRM database. Once you configure the external OData feeds, CRM surfaces this data to users as if it lives directly within CRM. After reading this and doing some digging into the new feature, I feel it offers a solid solution to common problems with data integration.
In the old days…
Prior to Virtual Entities, developers had options for external systems integration, but each required a bit of work to implement and maintain. Depending on the business requirements and and the sources of external data, integrating with external systems could easily add a significant effort to a project. Some common examples:
- Copy the data – Build CRM entities to host the data within CRM, and implement services to sync data between CRM and the external system. In this case, the data is housed within CRM and developers have control over the sync process, such as the ability for a one way, read-only sync or a full bidirectional synchronization between the systems.
- Retrieve Multiple Plugins – Build Retrieve Multiple plug-ins that surface data to end users as if the information lives within CRM, such as within a view or advanced find. This is a popular method for external systems integration as it looks to the end user as if the data is coming from CRM and not the external system.
- Client site integration – Build CRM web resources that surface data via client side API calls. This is a handy method for presenting end users with external systems information either through script or an HTML web resource, but all work is done on the client.
These are just three examples and I know many more are available. Any of these examples might be just fine for your solution, but a business requirements can quickly limit the use of one or all of these possibilities. For example, copying the data into CRM can quickly fail based on security limitations. What if the required data includes Personally Identifiable Information (PII)? While on a project for Veterans Affairs, the PII requirements immediately shot down our option to simply copy the data into CRM entities. The second option of retrieve multiple plugins is a popular option, but some research on the topic quickly shows frequent issues with system performance.
Now, Virtual Entities!
So, what is so special about this new feature?
Configuration is the first thing that stands out to me. Non developers can configure a Virtual Entity and present external data: configure a data source, its endpoint and parameters, then define the entity and fields, and you have external data available in your system. As business analysts and developers both know, this can be a major time saver for a project.
Standardization is the another benefit of using Virtual Entities. When reviewing external systems integration solutions on older CRM implementations, you may encounter a wide variety of implementation methods. This is not necessarily a bad thing since there is rarely a single solution for every problem. But you can often discover multiple methods implemented within a single solution. This creates a few issues for long term maintainability and just plain confusion within the project team. Virtual Entities can serve as a default for external data integration. This provides a standard, well documented and supported method for external systems integration, so as new requirements arise, the team all starts on the same page with the new design.
My thoughts immediately went to the implementation benefits of the feature. But the first benefit listed in Jesper’s post is a big one for end users. With Virtual Entities, end user access the Virtual Entity data as if the data resides within CRM, through search results, grids, reports, and dashboards. This is a huge benefit if your end users leverage reports and dashboards in their day to day operations.
Some usage examples
Simple examples for Virtual Entities that immediately come to mind involve reference data. How often have you implemented a list of US States as a custom entity and a lookup? It would be really handy to just point to a list of US States, Cities, or other address data elements provided by an OData feed hosted by the USPS.
More complex scenarios crop up around shared data within a single organization. I often think of the health care industry for integrated systems examples. A customer might have a CRM solution that manages post discharge scheduling for patients. Instead of replicating the patient data within CRM, you can now implement Patient as a Virtual Entity that pulls only required patient data for your scheduling system to operate. This allows the health care system to build out a powerful scheduling system while their patient information system remains the sole system of source for PII data.
Another interesting usage of Virtual Entities could be behind the scenes, retrieving data that is never directly surfaced to the end user. Imagine an insurance company that leverages regularly changing rates for calculations that is already well managed in another system. Instead of replicating this data in CRM or building services that periodically retrieve this data, we might be able to leverage a Virtual Entity that can be queried in plugins, workflows or reports as any other entity. This mean one less data set to manage and you would ensure that the latest and greatest values are being used real time.
Summing up
With each of these possible uses, serious testing and an overall architecture assessment should be taken into account. A project team should be asking some big questions before diving in and using Virtual Entities: How will using virtual entities impact performance, either on server side or client side processing? What are the limitations on the endpoint and will your system cause issues with overuse? These are just a few and I know those evaluating this feature have asked many more. But these kinds of questions should be asked when evaluating any new feature and when designing any new system.
Overall, this looks to be a powerful new feature for D365 CRM and a big step in showcasing it as a extensible line of business development platform.
As always, feedback is welcome. I am still learning about this new feature, so I would be very interested in hearing about some real world examples, some cool features that you have discovered, or some gotchas that might save us all from some hassle in the future.
Further reading
Below are just a few links right from the SDK that provide some additional reading on the Virtual Entities feature.
Create and edit virtual entities that contain data from an external data source
OData v4 Data Provider configuration, requirements, and best practices
This additional article by Jason Lattimer outlines how to create a custom Data Provider for Virtual Entities. This scenario allows repurposing existing data providers that may not meet all of the requirements for the out of the box Virtual Entities. Another great read…