File transfer

Most integrations that arrive in Integration Studio are file-shaped: a supplier drops a catalog on SFTP, an ERP writes a nightly CSV, a partner expects an export in a folder by 06:00. This page is the shape of that work.

The two node sets

Five operations each, over SFTP and over FTP or FTPS, plus an existence check in the control category:

The five file operations emit out and error. The existence check branches three ways — exists, missing, error — which is what makes it usable as a gate rather than as a failure.

Downloads are base64-encoded on the way out, and uploads take base64 or plain text, so a file passing through a workflow is a value like any other. Two nodes exist precisely so that large files do not pass through the workflow — see below.

The credentials

An SFTP credential (revenexx:sftp) or an FTP credential (revenexx:ftp), both static auth: the fields are the access data. Create one, test it in the dialog before saving, and every SFTP node in every workflow can reference it. See Credentials.

One credential per system, not per workflow. Three suppliers with three SFTP accounts is three credentials; one supplier read by four workflows is one.

The pattern worth copying

Every shipped file template is a variation on the same five steps, and it is the pattern to reach for:

1. List the inbox, optionally filtered by a glob pattern — *.xml, orders_*.csv.
2. Iterate the listing, so the rest of the flow runs once per file.
3. Do the work with the file: stage it into the import service, parse it, push it somewhere.
4. Move the file to an archive folder on success.
5. Move it to a failed folder on the error branch instead.

/inbound/          files waiting to be processed
/inbound/archive/  files that were processed
/inbound/failed/   files that broke, kept for inspection

Two properties make this worth the extra node:

• It is naturally idempotent. The second firing of a schedule finds an empty inbox, so a schedule's at-least-once delivery stops being a problem you have to think about.
• Nothing disappears silently. A file that broke is still on the server, in a folder whose name says what happened, next to a run whose timeline says why.

Prefer move to delete. Deleting the source file is one lost run away from a supplier having to re-send a catalog.

Waiting for a complete file

A file that appears in a directory is not necessarily finished being written. Two ways out, in order of preference:

• A marker file. The sending system writes catalog.xml and then catalog.ready. Gate the flow on revenexx:sftp-file-exists against the marker, and delete the marker when you are done.
• A staging folder on the sender's side. The sender writes elsewhere and moves the finished file into the inbox, so it appears atomically.

Neither is something the platform can do for you — it is a convention with the sending system. Agree it before the first delivery, not after the first truncated import.

Large files: stream past the workflow

A catalog can be hundreds of megabytes, and pulling that through a workflow as a value is the wrong shape. Two nodes exist for it:

• Stage From SFTP (revenexx:baseline-io-stage-from-sftp) streams a file straight from the SFTP server into the import service, however large it is, and emits the object key an import run needs.
• Deliver to SFTP (revenexx:baseline-io-deliver-to-sftp) streams an export result from its download URL straight onto an SFTP server.

Both take an SFTP credential and a Revenexx API credential with the io scope. This is why a 469 MB feed is an ordinary case rather than something to work around, and it is what the BMECAT template uses.

Raise the node's timeoutSeconds for a step like this. The default is 10 minutes and the ceiling is two hours; a gigabyte catalog import is a normal case for the ceiling. See Failure handling.

Where to go next

• Node catalog — every file node with its ports.
• Reaching Revenue Cloud — the import and export side of a file pipeline.
• Templates — the shipped flows that implement this pattern.
• Storage — the platform's own asset storage, for files that should become managed assets rather than pass through.