Create and manage SFTP files using SSH and Azure Logic Apps

  • Commodity
  • 14 minutes to read

To automate tasks that create and manage files on a Secure File Transfer Protocol (SFTP) server using the Secure Beat (SSH) protocol, yous can create automatic integration workflows by using Azure Logic Apps and the SFTP-SSH connector. SFTP is a network protocol that provides file access, file transfer, and file direction over whatever reliable data stream.

Here are some example tasks y'all can automate:

  • Monitor when files are added or changed.
  • Become, create, copy, rename, update, listing, and delete files.
  • Create folders.
  • Get file content and metadata.
  • Extract archives to folders.

In your workflow, you tin can apply a trigger that monitors events on your SFTP server and makes output available to other actions. You can then employ actions to perform various tasks on your SFTP server. You lot can also include other actions that use the output from SFTP-SSH actions. For example, if yous regularly retrieve files from your SFTP server, you tin can transport email alerts about those files and their content using the Function 365 Outlook connector or Outlook.com connector. If you're new to logic apps, review What is Azure Logic Apps?

For differences between the SFTP-SSH connector and the SFTP connector, review the Compare SFTP-SSH versus SFTP section later in this topic.

Limits

  • The SFTP-SSH connector currently doesn't support these SFTP servers:

    • IBM DataPower
    • MessageWay
    • OpenText Secure MFT
    • OpenText GXS
    • Globalscape
    • SFTP for Azure Hulk Storage
    • FileMage Gateway

  • The post-obit SFTP-SSH actions support chunking:

    Activeness Chunking support Override chunk size support
    Copy file No Not applicable
    Create file Yes Aye
    Create binder Non applicable Not applicative
    Delete file Not applicable Non applicable
    Excerpt annal to folder Non applicable Not applicable
    Get file content Yep Yes
    Get file content using path Yeah Yes
    Get file metadata Not applicative Not applicable
    Get file metadata using path Not applicable Non applicable
    List files in folder Not applicable Non applicable
    Rename file Not applicable Non applicable
    Update file No Not applicable

    SFTP-SSH actions that support chunking can handle files up to ane GB, while SFTP-SSH actions that don't support chunking can handle files up to fifty MB. The default chunk size is xv MB. However, this size can dynamically change, starting from 5 MB and gradually increasing to the fifty-MB maximum. Dynamic sizing is based on factors such equally network latency, server response fourth dimension, and then on.

    Y'all tin can override this adaptive behavior when you lot specify a constant clamper size to utilize instead. This size can range from 5 MB to 50 MB. For example, suppose y'all take a 45-MB file and a network that tin that support that file size without latency. Adaptive chunking results in several calls, rather that one call. To reduce the number of calls, you can effort setting a 50-MB chunk size. In different scenario, if your logic app is timing out, for example, when using xv-MB chunks, you can try reducing the size to v MB.

    Chunk size is associated with a connection. This attribute means you can use the same connection for both deportment that support chunking and actions that don't support chunking. In this example, the chunk size for actions that don't support chunking ranges from v MB to 50 MB.

  • SFTP-SSH triggers don't back up message chunking. When triggers request file content, they select only files that are 15 MB or smaller. To get files larger than xv MB, follow this pattern instead:

    1. Use an SFTP-SSH trigger that returns only file properties. These triggers have names that include the clarification, (backdrop only).

    2. Follow the trigger with the SFTP-SSH Go file content activity. This action reads the complete file and implicitly uses message chunking.

Compare SFTP-SSH versus SFTP

The following listing describes key SFTP-SSH capabilities that differ from the SFTP connector:

  • Uses the SSH.Internet library, which is an open-source Secure Shell (SSH) library that supports .NET.

  • Provides the Create binder action, which creates a folder at the specified path on the SFTP server.

  • Provides the Rename file action, which renames a file on the SFTP server.

  • Caches the connection to SFTP server for up to 1 hr. This capability improves functioning and reduces how often the connector tries connecting to the server. To set the duration for this caching beliefs, edit the ClientAliveInterval property in the SSH configuration on your SFTP server.

Prerequisites

  • An Azure account and subscription. If you don't take an Azure subscription, sign up for a gratuitous Azure business relationship.

  • Your SFTP server address and business relationship credentials, so your workflow can access your SFTP account. You also need access to an SSH private key and the SSH private key countersign. To upload big files using chunking, you need both read and write access for the root binder on your SFTP server. Otherwise, y'all go a "401 Unauthorized" error.

    The SFTP-SSH connector supports both private key authentication and password authentication. However, the SFTP-SSH connector supports only these private key formats, encryption algorithms, fingerprints, and key exchange algorithms:

    • Individual key formats: RSA (Rivest Shamir Adleman) and DSA (Digital Signature Algorithm) keys in both OpenSSH and ssh.com formats. If your private key is in PuTTY (.ppk) file format, first catechumen the fundamental to the OpenSSH (.pem) file format.
    • Encryption algorithms: Review Encryption Method - SSH.Cyberspace.
    • Fingerprint: MD5
    • Central commutation algorithms: Review Key Exchange Method - SSH.NET.

    After y'all add together an SFTP-SSH trigger or action to your workflow, you have to provide connection information for your SFTP server. When yous provide your SSH private key for this connexion, don't manually enter or edit the cardinal , which might cause the connectedness to fail. Instead, brand sure that you copy the primal from your SSH private key file, and paste that key into the connexion details. For more data, meet the Connect to SFTP with SSH section later this article.

  • Basic knowledge about how to create logic apps

  • The logic app workflow where you want to admission your SFTP account. To first with an SFTP-SSH trigger, create a blank logic app workflow. To use an SFTP-SSH action, commencement your workflow with some other trigger, for example, the Recurrence trigger.

How SFTP-SSH triggers work

Polling beliefs

SFTP-SSH triggers poll the SFTP file system and look for any file that changed since the last poll. Some tools let you preserve the timestamp when the files change. In these cases, y'all have to disable this feature so your trigger tin work. Here are some common settings:

SFTP client Action
Winscp Go to Options > Preferences > Transfer > Edit > Preserve timestamp > Disable
FileZilla Get to Transfer > Preserve timestamps of transferred files > Disable

When a trigger finds a new file, the trigger checks that the new file is consummate, and non partially written. For example, a file might have changes in progress when the trigger checks the file server. To avoid returning a partially written file, the trigger notes the timestamp for the file that has contempo changes, but doesn't immediately return that file. The trigger returns the file but when polling the server again. Sometimes, this behavior might cause a filibuster that is upward to twice the trigger's polling interval.

Trigger recurrence shift and migrate (daylight saving time)

Recurring connection-based triggers where you need to create a connection first, such equally the managed SFTP-SSH trigger, differ from built-in triggers that run natively in Azure Logic Apps, such as the Recurrence trigger. In recurring connection-based triggers, the recurrence schedule isn't the only driver that controls execution, and the fourth dimension zone only determines the initial offset time. Subsequent runs depend on the recurrence schedule, the concluding trigger execution, and other factors that might cause run times to drift or produce unexpected behavior. For example, unexpected behavior can include failure to maintain the specified schedule when daylight saving time (DST) starts and ends.

To brand sure that the recurrence time doesn't shift when DST takes upshot, manually adjust the recurrence. That way, your workflow continues to run at the expected time or specified beginning time. Otherwise, the start fourth dimension shifts one hr forrard when DST starts and one hour astern when DST ends. For more information, see Recurrence for connection-based triggers.

Convert PuTTY-based key to OpenSSH

The PuTTY format and OpenSSH format utilise dissimilar file name extensions. The PuTTY format uses the .ppk, or PuTTY Private Key, file name extension. The OpenSSH format uses the .pem, or Privacy Enhanced Mail, file name extension. If your private key is in PuTTY format, and you have to utilize OpenSSH format, first convert the primal to the OpenSSH format by following these steps:

Unix-based Bone

  1. If y'all don't accept the PuTTY tools installed on your system, exercise that at present, for example:

    sudo apt-get install -y putty

  2. Run this control, which creates a file that you can use with the SFTP-SSH connector:

    puttygen <path-to-private-key-file-in-PuTTY-format> -O private-openssh -o <path-to-private-key-file-in-OpenSSH-format>

    For example:

    puttygen /tmp/sftp/my-private-key-putty.ppk -O private-openssh -o /tmp/sftp/my-private-cardinal-openssh.pem

Windows OS

  1. If y'all haven't done then already, download the latest PuTTY Generator (puttygen.exe) tool, and then launch the tool.

  2. On this screen, select Load.

    Select "Load"

  3. Scan to your individual primal file in PuTTY format, and select Open.

  4. From the Conversions menu, select Consign OpenSSH central.

    Select "Export OpenSSH key"

  5. Salvage the private primal file with the .pem file proper name extension.

Considerations

This section describes considerations to review when y'all use this connector'south triggers and actions.

Use different SFTP folders for file upload and processing

On your SFTP server, use dissever folders for storing uploaded files and for the trigger to monitor those files for processing. Otherwise, the trigger won't burn down and behaves unpredictably, for instance, skipping a random number of files that the trigger processes. However, this requirement means that you need a way to move files between those folders.

If this trigger problem happens, remove the files from the folder that the trigger monitors, and use a different folder to store the uploaded files.

Create file

To create a file on your SFTP server, yous can utilise the SFTP-SSH Create file activity. When this action creates the file, the Logic Apps service also automatically calls your SFTP server to get the file's metadata. However, if you move the newly created file before the Logic Apps service can brand the call to get the metadata, you lot get a 404 error message, 'A reference was fabricated to a file or folder which does non exist'. To skip reading the file'due south metadata after file cosmos, follow the steps to add and prepare the Get all file metadata property to No.

Of import

If you use chunking with SFTP-SSH operations that create files on your SFTP server, these operations create temporary .partial and .lock files. These files aid the operations use chunking. Don't remove or alter these files. Otherwise, the file operations neglect. When the operations cease, they delete the temporary files.

Connect to SFTP with SSH

When you lot add a trigger or action that connects to a service or system for the first time, the workflow designer prompts you to create a connexion past providing the necessary information, which varies based on the connexion, for example:

  • The name that yous want to employ for the new connection

  • The proper name for the organisation or server

  • Your user or account credentials

  • The authentication type to use

  1. Sign in to the Azure portal, and open your logic app in Logic App Designer, if not open already.

  2. For blank logic apps, in the search box, enter sftp ssh as your filter. Under the triggers list, select the trigger you lot want.

    -or-

    For existing logic apps, under the final stride where yous desire to add an activeness, select New pace. In the search box, enter sftp ssh as your filter. Under the actions list, select the action you want.

    To add an action between steps, motion your pointer over the arrow betwixt steps. Select the plus sign (+) that appears, and then select Add together an action.

  3. Provide the necessary details for your connection.

    Of import

    When yous enter your SSH individual key in the SSH private primal property, follow these additional steps, which help make sure you provide the complete and right value for this property. An invalid primal causes the connection to fail.

    Although you lot tin can apply any text editor, here are sample steps that bear witness how to correctly re-create and paste your key by using Notepad.exe every bit an example.

    1. Open up your SSH individual key file in a text editor. These steps use Notepad as the case.

    2. On the Notepad Edit card, select Select All.

    3. Select Edit > Copy.

    4. In the SFTP-SSH trigger or action, paste the complete copied fundamental in the SSH private fundamental property, which supports multiple lines. Don't manually enter or edit the primal .

  4. Subsequently yous finish entering the connection details, select Create.

  5. Now provide the necessary details for your selected trigger or activeness and continue building your logic app's workflow.

Override chunk size

To override the default adaptive behavior that chunking uses, you can specify a abiding chunk size from 5 MB to 50 MB.

  1. In the action's upper-right corner, select the ellipses button (...), and and so select Settings.

    Open SFTP-SSH settings

  2. Under Content Transfer, in the Chunk size property, enter an integer value from 5 to l, for example:

    Specify chunk size to use instead

  3. Afterward you finish, select Washed.

Examples

SFTP - SSH trigger: When a file is added or modified

This trigger starts a workflow when a file is added or changed on an SFTP server. As instance follow-upwardly actions, the workflow can use a condition to bank check whether the file content meets specified criteria. If the content meets the condition, the Get file content SFTP-SSH action tin can get the content, and then another SFTP-SSH action tin put that file in a different folder on the SFTP server.

Enterprise example: You lot can utilize this trigger to monitor an SFTP folder for new files that stand for customer orders. Yous tin and so use an SFTP-SSH action such as Become file content and then you get the order's contents for further processing and store that order in an orders database.

SFTP - SSH action: Go file content using path

This action gets the content from a file on an SFTP server by specifying the file path. So for example, you tin add the trigger from the previous instance and a condition that the file's content must meet. If the condition is true, the activeness that gets the content tin run.

Troubleshoot problems

This section describes possible solutions to common errors or problems.

504 fault: "A connectedness effort failed considering the continued party did non properly respond later on a period of time, or established connectedness failed because connected host has failed to respond" or "Request to the SFTP server has taken more than than '00:00:thirty' seconds"

This error tin happen when your logic app can't successfully establish a connection with the SFTP server. There might exist different reasons for this problem, so attempt these troubleshooting options:

  • The connexion timeout is 20 seconds. Cheque that your SFTP server has good performance and intermediate devices, such as firewalls, aren't adding overhead.

  • If you have a firewall set upwardly, make sure that you add the Managed connector IP addresses for your region to the approved list. To find the IP addresses for your logic app'southward region, see Managed connector outbound IPs - Azure Logic Apps.

  • If this error happens intermittently, change the Retry policy setting on the SFTP-SSH action to a retry count higher than the default four retries.

  • Cheque whether SFTP server puts a limit on the number of connections from each IP address. If a limit exists, you might have to limit the number of concurrent logic app instances.

  • To reduce connection establishment price, in the SSH configuration for your SFTP server, increase the ClientAliveInterval holding to around i hour.

  • Review the SFTP server log to check whether the request from logic app reached the SFTP server. To get more than data virtually the connectivity problem, y'all can likewise run a network trace on your firewall and your SFTP server.

404 error: "A reference was made to a file or folder which does not exist"

This error can happen when your workflow creates a file on your SFTP server with the SFTP-SSH Create file activeness, but immediately moves that file earlier the Logic Apps service can go the file's metadata. When your workflow runs the Create file action, the Logic Apps service automatically calls your SFTP server to go the file'southward metadata. Still, if your logic app moves the file, the Logic Apps service can no longer discover the file then you get the 404 error message.

If you can't avert or filibuster moving the file, you can skip reading the file'due south metadata after file creation instead by following these steps:

  1. In the Create file activeness, open the Add new parameter list, select the Get all file metadata holding, and set the value to No.

  2. If you need this file metadata later, you tin can use the Get file metadata activeness.

Connector reference

For more technical details about this connector, such as triggers, actions, and limits as described by the connector's Swagger file, see the connector's reference folio.

Next steps

  • Acquire about other Logic Apps connectors