MuleSoft: Salesforce Synchronization with Retry Sample

In addition to MuleSoft connection Retry, this pattern adds the retry of rejected or failed records by the target system validation, in case another retry would succeed

Scenario

In a recent project I was working on, I needed to keep Salesforce objects updates synchronized with Siebel. The basic integration scenario is shown in Figure 1.

Figure 1: Integration Scenario

There is one complication though, sometimes Siebel will reject updates or new records as validations fails due to missing data or racing conditions with other data records. Rather than letting records fail and logging errors for support or the user to resubmit, a retry approach after a period of time could remedy most of the errors. As depicted in figure 2, the logic for the synch is as follows:

  1. The synchronization job is schedule to run every 10 minutes.
  2. Fetch the last datetime the synch had run (in case the synch job was stopped for any reason such as maintenance).
  3. Fetch all modified or created accounts IDs from Salesforce since the last run
  4. Process Accounts
    1. Fetch the Account Objects for the records ID from steps 3
    2. Do the necessary conversions to Siebel format and send the updates to Siebel
    3. Send the Siebel IDs for accepted records to Salesforce
    4. For records that got rejected by Siebel, keep track of the record IDs and store them in MuleSoft Object Store with the number of retries.
  5. If there are Stored Rejected Record IDs from a previous run in MuleSoft Object Sore
    1. Fetch the record ID from Object Store
    2. Execute Process Accounts step 4
    3. If the number of retries exceeds 3 times log the rejection and create a ServiceNow ticker so support can track and resolve the issue

Figure 2: Synchronization with Capturing errors and Retry

The Sample

In the sample code found at my GitHub (https://github.com/RefaatM/MuleSoft_SyncWithRetry), I am simulating Siebel with a MySQL DB. The implementation will synchronize the Salesforce Account changes with a table in MySQL DB. The next few sections, walks through the code details

Project Structure

When you open the sample code in AnyPoint Studio, you will find the project code consisting of:

  1. Src\main\mule which contains the 4 workflows
  2. Src\main\java which contains 1 java helper class
  3. Src\main\resources which contains the configurations config.yaml

The other packages are the standard packages of any MuleSoft project. The 4 Workflows, 1 Java class and config.yaml implement the solutions.

Figure 3: Project Structure

General Workflow and Global Configuration

Figure 4: General Workflow

Following the MuleSoft best practices the Global workflow is where you will find the Global Exception Handler in this sample it just logs the error (figure 4) and the global configuration to use a config file for the different environments and the connections to Salesforce and MySQL DB (figure 5).

Figure 5: Global Configuration

Figure 6 shows that the Default Error Hander is set to the “General Error Handler” in the General Workflow. For a refresher on MuleSoft Exception Handling see MuleSoft: Understanding Exception Handling/

Figure 6: Global Error handler Configuration

Figure 7 Shows the setting of config.yaml as the source for the configuration information. It is important not to hard code any environment or properties.

Figure 7: Configuration properties source file

The listing below shows the contents of the config.yaml . All sensitive information is masked with * to run the sample you will have to enter the proper information for your environment.

sfdc:


username: “**************”


password: “***********”


token: “*****”

db:


host: “localhost”


port: “3306”


user: “root”


password: “****”


database: “synchsample”

Figure 8 shows the configuration of the Salesforce connection with parametrized information to be retrieved from config.yaml

Figure 8: Salesforce configuration

Figure 9 shows the configuration of the MySQL DB connection with parametrized information to be retrieved from config.yaml

Figure 9: MySQL Connection configuration

MainFlow

Figure 10: MainFlow

The mainflow is triggered by the scheduler the logic is :

  1. Log the start time
  2. Call the Get Duration subflow
  3. Query Sales force for update accounts
  4. Convert the result to a list of IDs to query
  5. Call Record Processing subflow
  6. Log the operation is finished
  7. In case of any error just log it.

CallRecordsProcessing Workflow

Figure 11: CallRecordProcessing Subworkflow

  1. Check if the list of record IDs is empty
    1. Not Empty: call Process Records
    2. Empty: log no modified records found

GetDuration Subworkflow

Figure 12: GetDuration Subworkflow

  1. Check if Object store contains saved lastTimeRun
  2. Contains LastTimeRun:
    1. Fetch the lastTime
    2. Call Helper Java method to get the duration
  3. Does not contain LastTimeRun:
    1. Set the duration to 10 minutes

The Java method code is listed below which just calculates the duration since last time the workflow ran till now in minutes

package sfsynchwithretrypattern;

import java.time.Duration;

import java.time.LocalDateTime;

import java.time.ZonedDateTime;

public class StaticHelpers {

    public static long getDuration(ZonedDateTime lastTime) {

        LocalDateTime now = LocalDateTime.now();

     Duration duration = Duration.between(now, lastTime.toLocalDateTime());

     return Math.abs(duration.toMinutes());

    }

}

StoreRetryAccounts Workflow

Figure 13: StoreRetryAccounts SubWorkflow

  1. Check if there are stored RetryIDs
    1. RetryIDS stored: Append the new IDs to the existing IDS
    2. Does not Exist: set the store load to the new IDs
  2. Store the updated value

ProcessRecords SubWorkflow

Figure 14: ProcessRecords SubWorkflow

  1. Retrieve accounts with IDs
  2. For each retrieve account IF
    1. Check if the Account info exists in the DB
      1. Exists: update the account information
      2. Does not Exist: Insert a new record

    Note here I am taking an new records and saving them in the retry ID to simulate rejections. This should be moved to where you receive results and the errors

RetryProcessing Workflow

The retryprocessing workflow is triggered by a schedular, it starts with:

  1. Check if there are any retry id stored
    1. Exists: retrive the stored IDs and process them
    2. Does not Exist: Log no Stored IDs were found.

Conclusion

This is a good way to handle transient errors that occasionally occur during integrating system, when a simple retry might resolve the issue rather than having to ask the user to re-submit or getting support to chase the error. I hope this helps you in your projects. Feedback is welcome.

Figure 15: MySQL DB Table with Updates

Advertisements

MuleSoft: Understanding Exception Handling

I have been approached by several developers taking the Any point Platform Development: Fundamentals (Mule 4) training about the Exception Handling and the different scenarios in Module 10. The way it is described can be confusing. So here is how I understand it. Any MuleSoft flow like the one below

How this would be executed? Any Point studio and MuleSoft Runtime would convert it into a Java byte code. So it would be generated as a method or a function in Java. MuleSoft would put the code for this function within a Try -catch scope. If you have defined exceptions handlers in the error handling module it will emit code to catch and handle those exceptions. If there is not and there is a global error handler, it will emit code for the catch exceptions of the global error handler. Here is the thing that catches developers by surprise. If you have defined local handlers, then those cases are the only cases that would be handled and not the combination of the local cases and global error handler case, only one of them if there is a local error handler defined then that is it if there is not then the global error handler is emitted as catch cases

The 2nd point is On Error Propagate and On Error Continue options. If you chooser on Error Propagate then the coded emitted will throw the caught exception at the end of each catch. If you chose On Error continue then the Exception is not thrown. Think of it as if the code written below If you have been a Java , C#, C++. Or Python developer you should understand this basic programming concepts.


public void MianMethod(){
try {
OnErrorPropagate();
OnErrorContinue();
} catch (Exception E)
{
// default handler or global handler if defined
}
}
public void OnErrorPropagate() throws EOFException
{
try{
throw new EOFException();
}
catch( EOFException e)
{
Logger.getLogger(“ExceptionHandler”).log(Level.SEVERE,“Errot”,e.toString());
throw e;
}
}
public void OnErrorContinue()
{
try{
throw new EOFException();
}
catch( EOFException e)
{
Logger.getLogger(“ExceptionHandler”).log(Level.SEVERE,“Errot”,e.toString());

}
}
Hope this helps

MuleSoft: MCD Level 1 Mule 4 Certification Experience

OK Passed the Exam. How was it. I cannot tell you what the questions was and frankly I do not quite remember them. But here are my thoughts about the exam.

  1. You must go through the training course (which is free online, do all the exercises as there are concepts that are not in the slides or materials. And do the DYI exercises. I personally did them maybe 4 or 5 times. And I took the 3.8 online free course though that might confuse you as there changes between 3.8 and 4.0 like flowVars which is gone completely.
  2. I would recommend going quickly over the mule runtime documentation. And just trying to build a practical example.
  3. An understanding of Java and Spring either MVC or Boot frameworks would help but not necessary as it will help you understand how flows are translated into Spring code then compiled
  4. If you have time on your hands, go to git hub and download the free Sample for MuleSoft from MuleSoft and the open source code. Yes, that is an overkill.
  5. Now how about the questions, They are mostly tricky questions, most of the answers are extremely similar. You have to consider them thoroughly and find the best answer. Many are very simply minor colon, or semi-colon difference. Very low level syntax error so yeah if you have special glasses for the computer use them!.
  6. If you are taking the course from Home make sure you turn off your cell phone, phone etc. and do not speak to yourself. And yes you will not be able to use your big screen TV, you will be limited to the laptop screen.
  7. Hope this helps. I wish I can say it was easy I answered it in almost halftime with reviews, but I was really annoyed by the questions and every question is just trying to trick you.

MuleSoft: File:List Interesting Observation

Working with MuleSoft file connector, I was expecting that the File->List (https://docs.mulesoft.com/connectors/file/file-list) operation would return a list of fileInfo objects (you know path, size etc) but it actually it returns a list of the contents of the files in the directory. This seemed odd to me as the documentation states


“The List operation returns a List of Messages, where each message represents any file or folder found within the Directory Path (directoryPath). By default, the operation does not read or list files or folders within any sub-folders of directoryPath.
To list files or folders within any sub-folders, you can set the recursive parameter to


https://docs.mulesoft.com/connectors/file/file-list

Here is the sample I was working with

I was intending to put a read file operation in the foreach however that just gave me an error

Here is sample of the logged messages

That was a head scratch-er I thought I had done some mistake in the list parameters, but it seems that is how the file connector list operator works. Below you will see that part of the message for each fine the typeAttributes have the fileInfo information.

Implement Rest API in MuleSoft, Azure Logic Apps, Asp.Net core or Spring-Boot? MuleSoft: Step1- Defining an API in RAML

I have been working lately on comparing on comparing different technologies to build web API’s.. One of the main concerns was if we wanted to build a simple API service which technology would be easier, more productive to develop the service with. To provide a reference comparison I will build the same web service (in MuleSoft, Azure Logic App, Asp.Net Core, Spring Boot) and provide my notes as I go. The web service would provide the following functionality

  1. CRUD operations on an Authors entity
  2. CRUD operations on BOOKS entities where books are related to authors

All the Read (queries) should support:

  1. Filtering
  2. Searching
  3. Paging
  4. Supports Levels two and three in Richardson Maturity Model(see my previous post https://moustafarefaat.wordpress.com/2018/12/11/practical-rest-api-design-implementation-and-richardson-maturity-model/) . This means based on the Accept header of the request return the results as either:
    1. Pure JSON
    2. With HATEOUS

I will start with MuleSoft implementation.

Step 1. Define the API in RAML

With MuleSoft you get AnyPoint portal and you get the design center, which helps you designing the API RAML. There is API Designer visual Editor which can help you in the beginning.

 

 

 

Though it has many weakness such as:

  1. Once when you switch to RAML editor you cannot go back.
  2. You cannot define your own Media Types you have to use form the list.

 

To finalize the API definition in RAML I had to manually edit though the editor was helping to get started. Below is a fragment of the API in RAML (The full solution will be published on my GitHub https://github.com/RefaatM )

Notice in the RAML that I have defined two responses for the Get operation of the Authors Resources. Full RAML is at (https://github.com/RefaatM/MuleSoftRestAPIExample/tree/master/src/main/resources/api)

#%RAML 1.0

title: GTBooks

description: |


GTBooks Example

version: ‘1.0’

mediaType:

application/json

application/xml

protocols:

HTTP

baseUri: /api/v1.0

 

types:


CreateAuthor:


description: This is a new DataType


type: object


properties:


Name:


required: true


example: Moustafa Refaat


description: Author Name


type: string


Nationality:


required: true


example: Canadian


description: ‘Author Nationality ‘


type: string


Date-of-Birth:


required: true


example: ‘2018-12-09’


description: Author Date of Birth


type: date-only


Date-of-Death:


required: false


example: ‘2018-12-09’


description: Author Date of Beath


type: date-only

 


Author:


description: This is a new DataType


type: CreateAuthor


properties:


Id:


required: true


example: 1


description: Author Id


type: integer


Age:


required: true


maximum: 200


minimum: 8


example: 10


description: Author Age


type: integer

 


AuthorHateoas:


description: Author with Hateoas information LINKS


type: Author


properties:


Links:


required: true


description: Property description


type: array


items:


required: true


type: Link

 

 


Link:


description: Hateoas LINK


type: object


properties:


href:


required: true


example: /Book/10


description: URL Link


type: string


rel:


required: true


example: GetBook


description: Operation


type: string


method:


required: true


example: GET


description: ‘HTTP Method Get, PUT,..’


type: string

/author:


get:


responses:


‘200’:


body:


application/json:


type: array


items:


type: Author


application/hateaos+json:


type: array


items:


type: AuthorHateoas


‘304’: {}


‘400’: {}


‘500’: {}


headers:


Accept:


example: ‘application/json ‘


description: application/json or application/hateaos+json


type: string


queryParameters:


sort-by:


required: false


example: Example


description: sort by


type: string


filteryby:


required: false


example: Example


description: Property description


type: string

 

(.. to be continued)