The Microsoft® Sync Framework is the new framework and runtime for adding synchronization, roaming, and offline capabilities to applications.

It supports peer-to-peer scenarios, works with devices and services, and is agnostic of data types, stores, and protocols. In this article, I’ll cover the high-level vision for the platform as well as the enabled scenarios made possible by the framework for developers, ISVs, and OEMs.

Several years ago, Microsoft assembled architects and researchers from across the company to understand the limitations preventing the seamless flow of data across applications, devices, and services to allow consumers access to their data wherever they need it. Throughout the investigations, several observations quickly emerged:

  • Lack of generic synchronization technology lead to the creation of numerous single-purpose solutions.
  • Solutions tended to reinvent the wheel; often falling into the same pitfalls explored by others.
  • Solutions could not be bridged as they made contradictory assumptions about the synchronization methodology (i.e., relied too heavily on specific topologies, data types, stores or business logic).

It became clear that the only way to make progress towards enabling seamless “content flow” was to build a common framework with the following attributes:

  • Powerful. Able to solve the numerous hard problems related to interoperability.
  • Flexible. Able to be used in all endpoints in the sync-enabled ecosystem.
  • Simple. Easy to use in any architecture to compose new systems.

Only then could the framework support roaming and sharing scenarios for content such as PIM, audio, video, settings, and files/folders across any number of PCs, services, and devices-all directly via peer-to-peer sync or through any number of intermediaries such as PCs or services.

At the same time, more and more applications were looking to enable an Outlook-like cached mode of operation in which an application operates offline and periodically synchronizes with the server. Clearly, the common framework would need to excel at scenarios such as:

  • Caching and line of business (LOB) applications that can take data from relational databases and disparate back-end services offline.
  • Taking Web services and Web applications offline, providing richer experiences or simply network resiliency.

Project Goals

The Microsoft Sync Framework was created with the following goals in mind:

  1. Provide a common runtime for developers building synchronization solutions, allowing them to reuse a reliable and optimized code base capable of addressing the numerous subtle issues that developers run into when building caching, offline, sharing or roaming scenarios.
  2. Facilitate content flow across solutions, even if they utilize different protocols and stores by standardizing on the synchronization metadata.
  3. Simplify the actual development of sync solutions by providing domain-specific and end-to-end components for common scenarios such as syncing relational databases, file systems, lists, devices, PIM, music, video, etc…

The Microsoft Sync Framework Approach

A key tenet of the Microsoft Sync Framework is to bring existing applications and devices into the sync ecosystem by leveraging as much as possible from their existing implementations. This meant supporting synchronization between data stores ranging from large distributed Web stores backed by relational databases down to the file systems found on removable USB drives.

The framework would also have to support numerous data types and schemas (such as relational data or files), as well as various sync topologies (such as peer-to-peer meshes or hub-and-spoke). These requirements emerged prominently whenever multiple sync solutions were bridged in a heterogeneous mesh.

For these reasons the heart of the Microsoft Sync Framework consists of the common synchronization metadata model. By agreeing on only the metadata without forcing agreement on protocols or storage, the Microsoft Sync Framework lays the foundation for content-flow as well as a generic solution for synchronization, roaming, and sharing.

The metadata at the core of Microsoft Sync Framework is highly efficient and compact, yet it provides full support for correct multi-master synchronization (that is, allowing concurrent updates on multiple endpoints). The Microsoft Sync Framework runtime implements the algorithms required to work with the metadata, allowing endpoints to easily participate in the ecosystem.

To make it easier and more productive for developers building on Microsoft technologies to leverage these capabilities, Microsoft has started integrating the enabling technology into our next-generation flagships and platforms. For more information on Sync Services for ADO.NET and for an example of a Microsoft Sync Framework-enabled solution for rich synchronization and caching of relational databases, please see Steve Lasker’s article, Caching with SQL Server Compact and the Microsoft Sync Framework for ADO.NET in this issue of CoDe Focus.

Components of the Microsoft Sync Framework

The synchronization framework team designed the Microsoft Sync Framework as a componentized and layered architecture to allow developers to pick and choose only what they require to enable their scenarios and ensure that assets built on the framework could participate in the content flow ecosystem we’re working to grow.

Microsoft designed the Microsoft Sync Framework as a componentized and layered architecture to allow developers to pick and choose only what they require to enable their scenarios and ensure that assets built on the framework could participate in the content flow ecosystem we’re working to grow.

The Microsoft Sync Framework’s designers divided these components into the following logical layers, which are available to both managed and native code developers. (Note: Device builds for ARM, SH, x86, MIPS as well as Apple Mac support are in development and will be provided in a future CTP on MSDN).

  • Core Sync Runtime. Infrastructure that includes the algorithms to utilize the metadata as well as components to drive roaming, sharing, offline, and sync on behalf of applications.
  • Sync Provider Framework. Components designed to make it easier to expose data to the platform. This is effectively the plug-in model to the Microsoft Sync Framework by which developers can either configure existing Sync Providers or write their own.
  • Domain-specific components. Infrastructure to facilitate the rapid development of end-to-end solutions involving specific stores such as SQL Server, SQL Server Compact, FAT, NTFS, etc.

Core Sync Runtime

The core runtime contains the metadata services used by all of the Microsoft Sync Framework clients. Some of the features include:

  • Multi-master metadata representation and management, including conflict detection, custom and preset conflict resolution handling, change enumeration assistance, etc.
  • Conflict preservation management and representation of conflict resolution.
  • Support for change units (tracking changes at a property level) and consistency units (dealing with logical groupings of objects).
  • Handling of filtering changes and filter sync (both for items and for parts of items).
  • Recovery from a multitude of failure scenarios, such as tombstone cleanup, interruptions, network failures, etc.
  • Synchronization session management, such as cancellation, progress reporting, etc.

The Microsoft Sync Framework provides an out-of-the-box Synchronization Agent that can synchronize endpoints when requested. The endpoints are abstracted using the notion of a Sync Provider, which is responsible for exposing the capabilities of a given endpoint, store or protocol. Applications use Sync Providers for the stores they need to synchronize and developers use the Microsoft Sync Framework to create custom Sync Providers for virtually any type of data.

Core to the framework is support for Simple Sharing Extensions (SSE). The Microsoft Sync Framework natively supports endpoints that wish to interoperate using SSE extensions for RSS and ATOM feeds. Furthermore, the framework provides services for feed generation and consumption, including the requisite conflict detection and preservation. While the Microsoft Sync Framework offers RSS/ATOM support out-of-the-box, developers can extend this support to other formats as well. High level producer/consumer APIs help easily turn any “Microsoft Sync Framework” provider into an SSE-enabled endpoint!

Sync Provider Framework

The Microsoft Sync Framework supports several ways of writing Sync Providers. Each is intended to make the experience as easy and efficient as possible. In addition to full-featured (that is, “full participant”) providers that are capable of true peer-to-peer sync, the framework provides support for allowing legacy and existing devices and applications to participate in the content flow ecosystem. Specifically, the Microsoft Sync Framework Sync Providers support the following scenarios:

  • Partial Participants. Providers that simply store but do not understand most of the sync metadata (e.g. a USB keychain, legacy phone, media device). Despite being very easy to develop even on endpoints that do not host the Microsoft Sync Framework engine, these providers can participate in all multi-master content flow scenarios.
  • Simple Participants. Endpoints that lack the ability to detect changes and that lack the ability to store metadata. The Microsoft Sync Framework has services to allow data from these participants to flow within the ecosystem on behalf of a fully featured participant/provider.
  • Anchor-Based Providers. Support for providers that rely on a simple tick-count based enumeration mechanism for their change detection mechanism (e.g. timestamps, tick counts, etc.).

Storage Specific Components

In addition to the Microsoft Sync Framework-enabled Sync Providers in development across Microsoft, the Microsoft Sync Framework includes several components to simplify the development of offline, sync, sharing, and roaming scenarios using specific stores and protocols. These components include:

  • Relational Data Providers. Synchronization Services for ADO.NET 2.0, the next version of Sync Services discussed in Steve Lasker’s article, (“Caching with SQL Server Compact and Sync Services for ADO.NET” in this issue of CoDe Focus) will ship with SQL Server 2008. Sync Services 2.0 is built on the Microsoft Sync Framework aligning these two evolving sync platforms. It continues to dramatically simplify the caching of a remote or local database for scenarios such as line of business and branch office/point-of-sale scenarios, while providing peer-to-peer and other advanced sync capabilities. These providers take full advantage of the new change-tracking features in SQL Server 2008 and SQL Server Compact.
  • SQL Server Compact Metadata Store. A ready-to-use component for storing sync metadata such as versions, anchors, and change detection information. This component greatly simplifies the development of custom providers that do not have a natural place to store metadata.
  • File and Folder Sync Provider. A ready-to-configure provider capable of representing any Win32-compatible file system (e.g. FAT, NTFS, removable device). This provider handles challenges such as change detection on FAT volumes (including move and rename heuristics), name-name collision resolution, update-delete conflicts (including hierarchical update-delete), and the ability to preview a synchronization action.
  • ADO.NET Data Services Offline Provider. ADO.NET Data Services providers are currently being explored through prototypes that allow synchronizing data using REST-style interfaces for taking data services offline.
  • Plus many more providers currently in development across Microsoft!

Sync-Enabled Content Flow Scenarios

By removing the requirement that endpoints need intimate knowledge of one another in order to share or roam data, the Microsoft Sync Framework creates a sync-enabled occasionally connected ecosystem capable of providing:

  • Offline support for rich Internet applications. Ability to take a Web service or an Internet application offline with all changes synced back to any other endpoint without conflicts.
  • Calendar, Contact, Task List sync. Support for field-level sync such as “first name,” “last name” in an item-enabled rich PIM sync scenario.
  • Rich media sync. Ability to efficiently sync a full media library, or smaller subset, to the cloud or to devices either directly from another endpoint or through intermediaries. Ability to sync properties such as ratings and play count without transferring the entire media file.
  • Easy caching. The ability to easily cache a subset of the information from a remote or local store to support offline scenarios.
  • Sharing. Ability to have endpoints, such as devices, share data between any number of devices, clients, and services with edits possible at any node. This makes tethered or over-the-air sharing or roaming of media, PIM, files, and settings a breeze.

Conclusion

While this is an introductory, high-level overview of the Microsoft Sync Framework CTP1, you’ll find plenty of material including “How-to” documents, full-featured samples, articles, and white papers as we unveil the framework on MSDN and at conferences in North America and Europe.

Please visit http://msdn.microsoft.com/sync to get a copy of the Microsoft Sync Framework CTP1 SDK! I’m personally looking forward to reading your comments and feedback as we extend the framework and deliver on the content flow vision and strategy. I encourage developers to evaluate the SDK and request the features and improvements that will help you become more productive and the platform easier to use. We can be reached at syncsdk@microsoft.com and look forward to your feedback!