This article is relevant if you need to move files between Amazon S3 and NetSuite.
Background
The world continues to adopt cloud-based computing. A remote file storage cabinet is now common among all the large information platform providers. Each of them has a different approach to how they present file-based information.
In our NetSuite Systems Integration Practice, we are frequently in conversations with our clients to help them design and build integrations between NetSuite and any type of third party. File-based data migration has been a long-term, highly reliable, and widely accepted technique to move large data sets. By its very nature, it has built-in latency, which is valuable when designing loosely coupled bridged applications. Our Firm offers clients a great value by providing free of license charge tools to import and export complex NetSuite data sets.
Hence, file-based distribution tools are essential in system integration. While NetSuite offers an sFTP Connector (via the use of NetSuite’s underlying sFTP API library), we built one free for our clients that is more robust.
Prior to the availability of the sFTP API (N/sftp), we offered our clients a high-performance middleware approach. See my 2015 article, which discusses the open-source-driven Apache Camel framework. With the open-source framework, we easily access any data source. The approach, though, required standing up a service that then required monitoring and a degree of ongoing care. Prior to now, through this approach, we helped clients transport Amazon S3 based files to-and-from NetSuite.
One of our Fortune 500 clients using our Apache Camel middleware services to pass Amazon S3 data recognized that we had built an sFTP connector hosted right on NetSuite. They wondered, “if we can have S3 files move between NetSuite without the need for a middleware server, we would have fewer operational and security worries”. Thus, they challenged us to produce an S3 Connector that was modeled after our Prolecto sFTP Connector.
Introducing the Amazon S3 Connector for NetSuite
Given our client’s challenge, we prototyped the feasibility of moving files between S3 and NetSuite without the need for a middleware solution and other components. Using freely available software libraries provided by Amazon, we were able to adapt them to fit the NetSuite server environment. At that point, we needed to do the work to properly and securely build the Amazon authentication tokens. Once we pushed through those technical challenges, we now had all we needed to adapt our Prolecto sFTP Connector to become a Prolecto S3 Connector.
Key Features of the NetSuite S3 Integration Connector
Since our work in 2020 to build the Prolecto sFTP Connector, we have continued to enhance the capacities our clients demanded. Thus, we could bring those capacities into the Prolecto S3 Connector for NetSuite. Here are key capacities (click on the image for a better view):
- Profile Driven: we need to be able to set up a profile with the pointers for the target Amazon S3 Bucket and the local NetSuite file cabinet. Here we can specify search and file patterns to drive which files are processed.
- Bidirectional: we need to be able to send and receive files in either or both directions.
- Archive File Management: we need to have job control and thus move files to an archive folder on the remote and local file system after a successful file transfer.
- File Specifications: we need to be able to specify patterns for files to be picked up and for transfer. Using Regex and Saved Search can help us process files with complex file naming conventions.
- Scheduled Routines: NetSuite supports a scheduler that is granular up to every 15 minutes. If needed, we can have other events trigger immediate file transfers.
- Many Profile Considerations: in our sFTP profile system, we addressed a scalability concern using NetSuite Map/Reduce patterns when there may be 100s of profiles defined.
- Testing and Logging: when doing the setup, it’s important to see if connectivity is working. We need interactive buttons to drive on-demand execution with good messaging. We logged operations to help build dashboards to check for any errors and ensure trouble-free operations.
- Secure: the application is secure as it is hosted on NetSuite and uses HTTPS transport to move files between NetSuite’s and Amazon’s infrastructures.
Consideration for Binary Files
At the time of this writing, the Prolecto S3 Connector for NetSuite does not support binary files. The primary issue is due to the way we need to be prepared for receiving large (many understand the challenging NetSuite 10 megabyte+ file limitation) files using chunks of information. However, if we pass flat files, CSV files, XML files or other string-based files, we are good to go.
Note On Third-Party NetSuite File Systems
For organizations that are using S3 in many types of operations, it may make sense to move away from NetSuite’s File Cabinet altogether. In my 2020 article, Watch a Video to See How you Can Email Images to NetSuite and Tag them with Additional Attributes, I speak about the use of eXtendTech’s eXtendFiles tools that we frequently use in our client implementations due the way we can programmatically integrate files into our business process algorithms. Indeed, what we appreciate about their framework is the ability to connect uniformly to many different remote file systems, including S3. Thus, if a client is using these tools, we can approach the Binary file limitation from a different angle.
Get the Prolecto S3 Connector for NetSuite
Similar to all the algorithms we create for our clients, this bundle is available to all of our clients without a license charge. Our clients appreciate that we are in fundamental business logic discussions with no alternative sales motives, as we can bring all of our intellectual property and know-how to their challenge. We know that file distribution is just one essential component of many elements needed to produce an overall desired situation. Our commitment is to listen carefully and bring comprehensive Netsuite-driven strategies and tactics to a challenge to break through any perceived limitations. After all, we need to remember that “it’s just software.”
If you found this article relevant, feel free to sign up for notifications to new articles as I post them. If you need to connect NetSuite to Amazon S3 and would like to avoid recurring fees, let’s have a conversation.
