Async Tracking

Introduction

There are two main purposes to AsyncTracking. One is to allow faster tracking than with the synchronous tracking by making each tracking independent of the others, so that the tracking party can continue its work without waiting for trackings to complete. The other purpose is to allow tracking to continue during service windows; where trackings are stored temporarily in the offline database and transfered to Link’s database when it’s ready.

The first part, supporting faster tracking, changes the way that interchanges and documents are identified. In the synchronous tracking scenario the server generates identifiers and the tracking party has to wait for the identifier before it can generate the next tracking. With asynchronous tracking, the tracking partner assigns a guid to each interchange and document and use these to continue tracking without waiting. When trackings are persisted to the Link database identifiers are still generated as usual, but trackings are resolved using the guids.

The second part, continue tracking during service window, also changes the way things work. All asynchronous trackings are initially written to a new offline database and then subsequently transfered to Links database. This transfer is handled by the async tracking job, which runs every minute (configurable) and ensures that all trackings in the offline database are transfered. When Link is offline, trackings remain in the offline database until Links database becomes online again, at which point they are transfered.

To facilitate offline operations, it is necessary to synchronize (one-way) users, permissions, clients and redirect urls to the offline database. The synchronization jobs runs hourly (configurable) to ensure this. Having a copy of these data allows the identity server to continue to issue tokens when the Link database is not available. Issued access tokens are the same and continues to work when switching between offline and online. The identity server automatically switches between offline and online mode. When it detects that it cannot access Links database it switches to offline mode and start a background task that checks every minute if it can revert to online mode.

During a planned service window the scheduling service should be stopped and thereby both jobs will cease operation. For async tracking to continue working, the web api must continue to run (on at least one web server). It is possible to set a ‘service unavailable’ flag and this will cause all the other api’s to return status code 503 (service unavailable) while async tracking continues to work.

Offline database

The offline database is intended to run on the web server, probably using a SQL Server Express instance, but that is not a requirement. It may also be installed on the same database server as Link or maybe elsewhere, where there is a database server. The requirements are that each web server has its own database and that the database and database server remain online. Obviously, if the offline database is not operational, neither is asynchronous tracking.

Synchronization (Link → Offline database)

This job is in charge of synchronizing all relevant tables from Link to the offline database.

This job is run as part of the Scheduler Service.

It is important that only one instance of this job runs per offline database.

  <appSettings>
    ...
    <add key="configConnString" value="SERVER=tcp:ebv-sql05\dev;TIMEOUT=5;DATABASE=Link;Integrated Security=SSPI" />
    <add key="configConnOfflineDbString" value="SERVER=.\SqlExpress;DATABASE=Link.API;Integrated Security=SSPI" />
	<add key="JobConfiguration:EnableOfflineDbSyncJob" value="false" />
	<add key="JobConfiguration:OfflineDbSyncJobTriggerIntervalInHours" value="1" />
  </appSettings>

There are four configurations to keep in mind when setting up the OfflineDbSyncJob.

configConnString is the connection string for the Link database. This is required.

configConnOfflineDbString is the connection string for the offline database. This is required.

JobConfiguration:EnableOfflineDbSyncJob is used to either enable or disable the job in the scheduler service. If this is not set, it will default to false.

JobConfiguration:OfflineDbSyncJobTriggerIntervalInHours is used to determine how often the job will run in hours. If this is not set, it will default to 1 hour.

IdentityServer online/offline

Async tracking job

This job is in charge of resolving all the async tracking requests that have come in from the Link REST API.

This job is run as part of the Scheduler Service.

It is important that only one instance of this job runs per offline database.

In a load balanced scenario, this job can run on each server. Each server will have their own offline database, but all async tracking jobs will resolve their requests to the same Link database.

The job prioritizes requests in the following order:

1. Create document or interchange

2. Upload document or interchange

3. Everything else

This order minimizes the chance of requests failing.

If failures do occur, the requests are left in the offline database so they can be retried later. An increasing delay is added to the request each time it fails so they do not bloat the next batch.

  <appSettings>
    ...
    <add key="configConnString" value="SERVER=tcp:ebv-sql05\dev;TIMEOUT=5;DATABASE=Link;Integrated Security=SSPI" />
    <add key="configConnOfflineDbString" value="SERVER=.\SqlExpress;DATABASE=Link.API;Integrated Security=SSPI" />
	<add key="JobConfiguration:EnableAsyncTrackingBufferSyncJob" value="true" />
	<add key="JobConfiguration:AsyncTrackingBufferSyncJobTriggerIntervalInMinutes" value="1" />
	<add key="JobConfiguration:AsyncTrackingBufferSyncJobRequestBatchSize" value="100" />
	<add key="JobConfiguration:AsyncTrackingBufferSyncJobMaxRequestRetentionInHours" value="24" />
  </appSettings>

There are six configurations to keep in mind when setting up the AsyncTrackingBufferSyncJob.

configConnString is the connection string for the Link database. This is required.

configConnOfflineDbString is the connection string for the offline database. This is required.

JobConfiguration:EnableAsyncTrackingBufferSyncJob is used to either enable or disable the job in the scheduler service. If this is not set, it will default to false.

JobConfiguration:AsyncTrackingBufferSyncJobTriggerIntervalInMinutes is used to determine how often the job will run in minutes. If this is not set, it will default to 1 minute.

JobConfiguration:AsyncTrackingBufferSyncJobRequestBatchSize is used to determine how many requests are retrieved from the database to be processed in a single loop. If this is not set, it will default to 100 requests.

JobConfiguration:AsyncTrackingBufferSyncJobMaxRequestRetentionInHours is used to determine when async tracking requests can safely be deleted from the database. If this is not set, it will default to 24 hours.

Automatic database cleanup (offline database)

The AsyncTrackingBufferSyncJob handles cleaning up the offline database.

During the process of finishing a batch, any request that is deemed to be too old, will be logged and deleted.

This limit can be set by the JobConfiguration:AsyncTrackingBufferSyncJobMaxRequestRetentionInHours configuration and will default to 24 hours if it is not set.

Service unavailable

In Service unavailable mode all api’s except asynchronous tracking returns status code 503, which indicates that the service is not currently available. Service unavailable mode is enabled by toggling the app-setting with the same name in the rest api’s web.config file. This must be done manually (both ways). It is meant to be a way to keep asynchronous tracking online (in offline mode), while all the synchronous api’s becomes unavailable, fx. during a service window.