How to use data transform in Pega?

Introduction

In this post we will see how effectively, we can use a Data transform rule.

Data transform is a rule helps in manipulating the data.

  • Remember data can be stored in either memory or it can be persisted in database.
  • Data stored in JVM memory are in the form clipboard pages and are temporary.

What do you mean by temporary here?

  • It means, when the server restarts all the clipboard pages in the memory get cleared.

Normally we use clipboard pages to support  processes.

For example – take amazon website.

  • When a customer purchase a product, our Pega application create a purchase request case at the back end.
  • Here we store all the case related details in database (say work table).

Normally why do we store some data?

  •  Yes we can use it for future reference. So we store the case details in DB for future reference.

Let’s come back to the Amazon purchase request case. You need to select a courier vendor to service the customer.

You use a Connect-REST to get the list of vendors available for the product – say you got 5 vendors in a page.

Do you think you need to save all the VendorsList in database?! No. Not required.

Those data are needed only at the time of processing and can be cleared. In this scenario, you save the data only in clipboard memory and you don’t persist the data to DB.

There may be some situation, where you need to update the data in the clipboard.

Pega provides us with a rule called ‘Data Transform

You can get the picture from the rule name. Help is transforming or processing data in the clipboard 🙂

Note: You can never persist any data using Data transform rule.

What is Data transform rule?

  • It belongs to data model category.
  • It is basically mapping data or process data in some target page. The source can be of any page data or just some hard-coded values.
  • It helps in data conversion and transformation

Always try to use data transform, when you want to just process the data (no persistent)

Data transform is very easy to understand 🙂

  • Always remember, Property rule in Pega helps to store the data. Properties are embedded to a page.
  • In Data transform rule, we transform the property values.

Let’s straight away create a new data transform rule.

Records – > Data Model -> Create a new data transform

In the data transform rule form, you have 2 main tabs.

  1. Definition
  2. Parameters

Definition tab

You can see 4 columns in Definition tab.

a) Action – You can take various action to support processing the data.

b) Target – You can specify a target page or property here.

c) Relation – This vary based on the action we perform. We will see that soon 🙂

d) Source – You can specify a source page or property. It can also contain some hard-coded values.

Throughout this post, I am going to explain you how to process data from one page to another page.

What are the pre-requisites?

  1. Create the following properties.

Page List property – SourceList, TargetList

Single value property – PolicyName

Value list property  – PhoneNumber

Value group property – EmailAddress

  1. Create a test activity and populate a page ‘ActivePage’.

  • Here I populated SourceList page list property with two pages – SourceList(1), SouceList(2)
  • In each page SourceList(1) & SourceList(2), I populated the PolicyName, PhoneNumber and EmailAddress values.

3. Create a new data transform – CheckDT

4. In the activity step, Use Apply-DataTransform method and refer the CheckDT data transform

Note: we can pass the parameters to the data transform

  • In short, I created a new page called – ActivePage in the clipboard and set some properties in the SourceList page list property.
  • I am using data transform to process the data.

Actions

1. Set – This is simple and most commonly used. We use this to set any data.

For example – I need to set the PolicyName in ActivePage or I need to set the CurrentDateTime value

Let’s see how we can do it.

Note: I didn’t use ActivePage.PolicyName, since ActivePage is the primary page for the data transform from the activity call.

  1. Set the PolicyName to hardcoded value “Automobile”
  2. Set pyAgeFromDate property value to CurrentDateTime
  3. Set the PhoneNumber(1) valuelist property to SourceList(1).PhoneNumber(1) valuelist property

Remember Source – SourceList is populated from the activity step.

Note: For valuelist and valuegroup properties, you can directly set the values without using subscript. I mean you can use .PhoneNumber = .SourceList(1).PhoneNumber, to copy the entire valuelist content.

  • Now run the activity and check if the values are populated in the ActivePage.

  • Click on the PhoneNumber valuelist property to check the values.

Looks good 🙂

2. Remove

  • This action help in removing any top level page or property values in the clipboard page.

Note: Removing is totally different from making property value to null

For example – If we don’t need the data in SourceList pagelist property, we can directly remove the property from clipboard.

  • Relation and source column are of no use

Run the activity and check in the clipboard.

  • You can see the SourceList populated from the activity is removed from the ActivePage.
  • Here you can also remove the top level page too.

I am removing the entire page from the clipboard.

Note: This is similar to Page-Remove method in activity.

You can see the page is removed

3. Update Page

  • This action helps in updating any page in the clipboard. You can use the relation column to choose the source page.
  • Select relation = ‘with values from’ to select a source page. You can also leave it blank.

You can see a Loop started. You can go on and update any property in the ActiveTestPage

  • Here I am going to copy the entire pagelist property from  from ActivePage.SourceList to ActiveTestPage.TargetList.

Note: If you want to update primary page, you no need to use this action.

Run the activity and check the clipboard.

  • You can see the SourceList value from ActivePage is set to TargetList in ActiveTestPage .

4. Apply Data transform

  • This action is used to cascade a new Data Transform.
  • You can call another data transform using this action.
  • Relation & source column are of no use.

You can click on the settings icon to control the parameters we pass to the new data transform.

We can either pass the parameter page or supply the required parameters.

I added a Data Transform – Initialize.

  • Open the activity and run the tracer.

You can see our Data Transform – CheckDT calls the Initialize data transform.

Using this action, you can reuse any existing data transform

5. Sort

  • This action is applicable only for PageList property type.

For example – You hit a Connect-REST service and get the PolicyList from different system.

  • The service returns the data un-ordered. You need to sort it based on Policy name and display it in section. Here you can use Sort action.

For testing, I modified the test activity to present the data in SourceList un-ordered.

Let’s sort the pagelist in our Data transform.

  • Specify the pagelist property – SourceList in the target field.
  • Click on the settings icon and add the Sorting properties.

Note: you can add multiple properties and the order takes the priority in sorting.

Here I am sorting using Policy Name. Run the activity and check in the clipboard.

You can see the pagelist property – SourceList reordered using PolicyName.

6. Comment

  • This action is Kind of dummy 😀 .
  • You can use to include some comments that other developers can understand what is happening in the data transform.

It never affects any property. At run time, Pega ignores this action.

7. When, Otherwise When, Otherwise

  • You can use these actions to conditionally invoke other action.
  • I mean you can invoke the actions based on some when conditions.

For Example –

When ( PolicyName = “Home”)

{

IsHomePolicy=’true’

}

Otherwise When ( PolicyName = “Auto”

{

IsAutoPolicy=’true’

}

Otherwise

IsOtherPolicy=’true’

You need to implement the above logic. You can make use of the When, Otherwise When conditions to achieve this.

  • You can see I used 3 conditionalized actions.

For testing I set PolicyName=”Umbrella”.

Run the activity and check in the clipboard. ‘IsOtherPolicy’ property value must be true.

Yes It is 🙂

  •  You can conditionally perform any actions.
  • Nested when conditions are also applicable.

8. Append to

Before this, let me refresh you with Page & PageList relation.

  • Think about a book. Book contain many pages. A collection of pages can be a book.

  • Like the same in Pega, a collection of pages form a page list.

Say we have a PageList – PolicyList and contain 2 results – PolicyList(1) & PolicyList(2)

PolicyList(1) is a separate page, similarly PolicyList(2) is a separate page.

  • Collection of pages form a page list property.

PolicyList(1) + PolicyList(2) = PolicyList

Now tell me what is dictionary meaning of APPEND?. We are inserting right.

  • Say you have a Pagelist property with 2 results ( 2 pages)- PageList(1) & PageList(2). When you append a new page to the pagelist property, then you can see the page in 3rd result – PageList(3)

Hope you will understand easily going forward 🙂

  • ‘Append to’ action helps in appending a page to a page list property.

For example: I have a Page list property – SourceList contains 2 Pages. I need to insert a new page to the page list. You can use this action.

Let’s check it.

In relation column, you have 4 options.

a) a new page

No explanation. Only testing 🙂

Step 1: In the activity append 2 pages to the SourceList property as shown below.

Step 2: In the data transform, Use ‘a new page’ in Relation.

Step 3: run the activity and check the clipboard.

You can see a new page – SourceList(3) added, but no values. Yes it is a new page

b) an existing page

Step 1: In the test activity, add a new step to update a page ‘ActiveTestPage’  with Gender = Male

Step 2: Update the data transform

Relation – An existing page.

Source – you can specify the newly created existing page ‘ActiveTestPage’

  • Here we are going to append ActiveTestPage page with SourceList property.

Step 3: Run the activity and check the clipboard.

You can see a new page appended with values from ActiveTestPage.

c) Current source page

What is current source page?

I will open the tracer for you. You need to check it. 😉

  • Open the tracer, run the activity.

So ActivePage.SourceList(3) is appended with which source page? – Active page. Which is the primary page context here.

You can verify it in clipboard.

d) Each page in

Till now we saw appending a single page to a pagelist property. What about appending a target pagelist with source pagelist.

Step 1: Update the data transform.

Step 2: run the activity and check in the clipboard.

You can see source pagelist is appended with target pagelist.

Note: For Append to action – both the source and target should be in same class.

9) Append to and Map to

First let’s see the difference between Append to vs Append to and Map to.

  • The target and source can be of different applies to class.
  • Append to and map to create a new page without any embedded properties, while Append to action creates a new page with all the values from source page.
  • You will have a child set action to perform with Append to and Map To

Note : Both action don’t work for page group properties. Only for pagelist property.

We use this mainly to copy data from one page to another page ( different applies to class)

  • Relation column is same as Append to action. You can see loop for child set action.
  • You can set properties using child set action.

First let’s test it.

Step 1: Update the data transform.

Step 2: run the activity and check in the clipboard.

You can see the pages appended from source without any values. The value we set in data transform is alone reflecting

10) For Each page & Exit for Each

For each page – You can use this action to loop a page list or page group property and update the values.

Exit for each – This action exit the remaining iterations in the loop.

Note: Exit For Each is not same as exit iteration you use in activity transitions.

  • Here in the above example, when Policy name = “Home”, we exit  remaining iteractions using Exit for Each action.

You can see an option – ‘Also use each page as source context’ in the source column! Why do we use it?

From the name you can say 🙂

For example – You need to update properties within the page – Means target and source are of same page.

  • In SourceList(1) you have PolicyType = “Home”. If you need to set the PolicyType value to PolicyName in SourceList(1).
  • In this case you can check the option and set directly.

In some scenario, you need to set PolicyName to property available in the class, from where we call the data transform ( primary page). here you can uncheck and make the Source column page context as primary context. 🙂

Open the tracer & run the activity.

  • You can see when PolicyName=”Home”, Exit For each occurs and the remaining Iteration stops.
  • Only SourceList(1) is iterated.

12) Exit Data transform

  • This action, ends the data transform abruptly. The other steps below are not executed.

  • Let’s check if step 2 is getting executed.

Open the tracer and run the activity.

  • You can see the data transform ends, when it encounters Exit data transform.

All actions over  Pardon me If I dragged some much

One more key area is Calling SuperClass data transform

First think of what is Super class?

  • Super class is parent class. Your mind might me thinking about inheritance now right 🙂

To check on the class inheritance path

App explorer -> Open the class -> right click -> Inheritance

  • You can see the class path till @baseclass.

Call superclass data transform

  • When this is checked, Pega checks the data transforms with the same name in parent class.

Say in my scenario, my Data Transform ‘CheckDT’ is in ‘TVS-TeslaVeh-Work’ class.

Here Pega checks the data transform in the parent classes.

Say it found two data transforms in same name CheckDT in ‘Work’ & ‘@baseclass’ ( Call superclass DT in work checked)

Here Pega executes the data transform in the following order

@baseclass  – > Work- -> TVS-TeslaVeh-Work

Let’s test it

Step 1: Create a new data transform in one of its parent class, say ‘TVS-TeslaVeh’.

I set pyLabel = ‘SuperClass’

Step 2: Check the call superclass data transform in CheckDT data transform in TVS-TeslaVeh-Work class.

Step 3: Open the tracer and run the activity.

You can see both the data transform gets executed, first parent (super class) then child.

 

When do we use this?

a) Mainly used when you create a new work item. You can see pyDefault data transform gets called in many classes.

This is because, you need to use some OOTB data transform ‘pyDefault’ in Work-, @baseclass to set some default values

I am going to trace the process, when I create a new purchase request case.

You can see the data transforms. Let’s go from the bottom.

  1. @baseclass  pyDefault – OOTB Data transform
  2. Work- pyDefault           – OOTB Data transform
  3. Work-Cover- pyDefault – OOTB Data transform
  4. TVS-TeslaVeh-Work-ServiceRequest pyDefault

First Let’s check the pyDefault in ‘TVS-TeslaVeh-Work-ServiceRequest’ class.

You can see the Call superclass data transform is checked 🙂

Let me show you the pyDefault in Work- class. OOTB data transform

  • You can see some standard properties like pyLabel, pyFlowName etc are set.
  • You can override the values in the data transform  specialized in class ‘TVS-TeslaVeh-Work-ServiceRequest’

 

b) When you use solution frameworks in your application, Your class structure will be inherited from solution framework class structure.

  • Here you may need to make use of data transforms  ‘pyDefault’ in solution framework classes.

End of data transform post 🙂

Catch you all in my next post……. Very soon 🙂

 

 

47 thoughts on “How to use data transform in Pega?

  1. Hello Prem,

    i would like know how we can replace Data Transform with Activity.My scenario is there is member table in pega database
    am using activity to fetch the data using obj methods , can we use data transform to fetch the data from database table

    1. No Pradeep, you can’t. As I’ve mentioned in the introduction part, data transforms help in playing with data only within JVM memory (clipboard pages). You cannot access DB or persist any data using DT.

      1. Hi ravi,
        You need to make use of conditional when to skip and logic and return to next iteration. I the sad part is we don’t have any exit iteration action like activity transitions.

        Note: Exit for each end all the iterations and exit the loop

  2. Hi Prem,

    I have few questions on Data Transforms
    1. How to handle exception handling in Data Transforms if we refer Data Page that throws exception
    2. If i have checked call superclass option in DT called A and for suppose i have 3 classes top to bottom A, B, C and all the three holds DT with same Name A , Now as per explanation DT gets executed in C, B, A now my actual question is how can i skip B and execute only C and A ?
    3. Remove is like a Page Remove , Can i Create a New Page using DT?

    1. 1. Data transforms in data page can set the response error message. They cannot set error on pages. You can just set the properties.
      For example, you used Connect-SOAP in data page. You can use data transform to set the capture the response and error values.
      If you want to set the error message to the data page, then you can use post load processing activity to set the error message.

      2. My suggestion, you can skip B by the withdrawing the rule in Class B.

      3. Yes, you create new page in many ways.
      Update page, Append – you need to refer the page in pages and classes tab.

      Hope you are clear now

  3. what is significance of the checkbox which is just beside the For Each page in step(i.e use as a source page context)

  4. Hi Prem,

    i have few queries on Data Transform
    1. Can we do a exception handling in DT eg. refering DataPage throwing exception?
    2. Remove is for Page-Remove as u said , can i create a new page using DT?
    3. if we select call superclass and DT is available in A, B ,C classes then suppose order of execution of DT is C,B, A then how can i skip the execution of B after C and continue to A?

    1. 1. Data transform exception handling – You cannot achieve the same like activity exception handling, but you can use some properties and compare values and set some error message value.
      2. Yes, you can. You can refer a page in data transform actions like update page, append into new page and refer the page in pages & classes tab. If the page is not available in clipboard, then a new page will be created.
      3. You need to withdraw the rule in class B

  5. Thanks a lot for giving us such a wonderful clarity on each and every topic you had posted. This indeed gives us a much needed theoritcal as well as practical knowledge to all of us. Please continue with your good work. 🙂

    One suggestion:-

    Is it possible for you to add one more category in your posts such as “references” which gives us the information about where this rule can be triggered/called from.
    For example:- DT can be called from an Activity, flow, DT itself, from Dynamic layout actions tab etc…??

    Because, At times, we tend to forget these small things.

    Waiting for your next post!!!!

  6. Great Work, there is no such Sufficient info on any website…………
    All Topics are clear and easy to understand, you are helping lot many people with this posts.

    Thanks Prem…………

    1. Oh, Navin!! Thank you so much for your appreciation. 🙂
      No words can tell how happy I feel on reading your comments. 🙂
      It is my pleasure to post stuffs, Navin and you are most welcome. 😀

  7. Hi Prem,

    Your post are so useful.How can i get all topics in pega from you.
    Because i missed lot of topics and i dont have much experience in this but when i am seeing your posts i understood easliy.
    Please send me all the topics to my mail.:)

    1. Hi Satya,

      Happy to hear that you find it useful. You can see all the topics in the site itself. You can find all the posts in the categories column.

  8. Hi Prem,
    Very nice post…Will you please explain how data transform help in data conversions and its format while mapping it from source to destination… Is it something like another programming language where we can change data types of variable(properties in pega).

  9. Hi Prem,

    What you mean by Solution framework?

    CPM framework and smart disputes are the example of solution frame work?

Leave a Reply

Your email address will not be published. Required fields are marked *