Considerations for NetSuite Map/Reduce Processing

This article is relevant if you are using NetSuite Map/Reduce scripting technologies.

Background

In our work assisting clients with their innovations, we often encounter a general pattern that requires that we work with records in a batch. NetSuite’s platform has evolved to help you work with batches of information. The general pattern is to get a list of database rows that then are individually acted on.

NetSuite’s Map/Reduce technology, which was introduced with the advent of the SuiteScript 2.0 platform, opened a more scalable approach for attacking batch processing patterns in the platform. With Map/Reduce, at a minimum, you effectively return to the starting function an array (dictionary) of IDs in the Input processing stage.  However, the data can be more complex which can be valuable at different points in the process. Then, the Map stage will receive a single event from each ID or array element that was provided in the Input stage. In the Reduce stage, if the data can be grouped, then you will receive a single element for each group (note, if the Input data was a list of unique IDs, then the same information supplied in Map stage will be the same data in Reduce stage). Finally, the Summary Stage brings it all together allowing you to review if there were errors and collect other statistical information.

In using the technology for a few years, our team has produced a Utilities Bundle that we give to our clients to help get more out of the Map/Reduce patterns we have developed. Along the way, we have noticed a couple important considerations that we think the community should understand. I would like to thank Boban D., one of our senior consultants, for providing detailed thinking.

Passing Data from the INPUT stage to the SUMMARY stage

One of the frustrating considerations about using Map/Reduce scripts is that they lack a facility to pass data from the INPUT stage to the SUMMARY stage.

For example, if the INPUT stage is working with a saved search that does not return a row or assume that every row that gets processed in the MAP stage has a failure, then nothing is ultimately passed to the final SUMMARY stage. If you are using tools (custom records) to track the initiation of your batch jobs (as we do in our Record Import / Export Manager processing tool to create scalable, handsfree record update automation), the SUMMARY step does not know which batch job started the whole process; consequently, there is no built-in way to then update the batch job record to update status or other information.

Note, readers may want to see this related article, NetSuite Script Pattern to Pass Parameters to Map Reduce Functions.

Solving the Missing Summary Stage Data

Our consultant solved this challenge by doing the following. The INPUT step can write a “payload” which can then be read by any of the following steps (including MAP, REDUCE, and SUMMARY).

The “key” used to write this payload is the script deployment id – because that is the only guaranteed, unique piece of information about the script, which every stage can access, without having to pass it from one step to the next. Even if a script has multiple deployments running simultaneously, they each have a unique deployment ID.

The method then is to write your “payload” in the INPUT stage to a data store, where the deployment ID is the “key” … and then at any other stage, you can read using the same key. The “payload” can be anything meaningful to the process.

Our consultant solved this and then added it to our Utilities Library   Here is what the pattern looks like.

Create a simple custom record with only two fields:

1. Name: The “name” (key) is the deployment ID
2. Data: The “data” is the payload

To make it easy for scripts to use this, the consultant created two new functions in a library:

1. writeToScriptDeploymentCache(data): call this method (in the INPUT stage, usually) with the data you want to preserve. It automatically writes it using the current script’s deployment ID as the key
2. readFromScriptDeploymentCache(): call this method (in any other stage) to retrieve the data you saved earlier

In the end, all you need to really know/do in your scripts is this:

1. In the INPUT stage, you just call writeToScriptDeploymentCache and pass it the data you want to preserve
2. In any other stage, you call readFromScriptDeploymentCache() to retrieve the same data

With this pattern, you have all you need to know to update a batch job with a status or other information.

Missing NetSuite Map / Reduce Information

We believe there is a challenge in the platform and have hope that it will be resolved. Here is what we have observed.  Imagine that your Map/Reduce INPUT stage is using a saved search like this

return search.create({
	    type:       "customrecord_xxx",
	    filters:    [ ["isinactive","is","F"] ],
	    columns:    ["internalid”,”custrecord_vendor_list"]
	});

Let’s say that on this custom record, the field “custrecord_vendor_list” is a MULTI SELECT field of type “Vendor”

If a row of the custom record has a Vendor (in the multi-select field) whose name contains a comma, then for that row of the saved search, NetSuite will not return column “custrecord_vendor_list” at all.  In other words, the row being passed to the Map step will look as if it didn’t include “custrecord_vendor_list” in the requested columns. Without this information, your logic may incorrectly branch.

Work with Expert NetSuite Professionals

My hope with this article is that you deepened your understanding of NetSuite batch processing challenges with Map/Reduce technologies. If you found this article valuable, feel free to receive notifications of new articles.

If you are looking to work with NetSuite professionals that have demonstrated accomplishments, or you would like to be part of a team that holds high standards for care and leadership, then let’s have a conversation.

Marty Zigman

Holding all three official certifications, Marty is regarded as the top NetSuite expert and leads a team of senior professionals at Prolecto Resources, Inc. He is a former Deloitte & Touche CPA and has held CTO roles. For over 30 years, Marty has produced leadership in ERP, CRM and eCommerce business systems. Contact Marty to set up a conversation.

More Posts - Website - Twitter - Facebook - LinkedIn - YouTube