HTTP Download

Author: Kostas Tsakiridis, 2BrightSparks Pte. Ltd.

Starting with SyncBackPro V11, users can backup/mirror files directly from web servers using a new location type called HTTP Download which can be used as a source for backup and mirror profiles. Sync profiles are not supported when using this type of location since HTTP Download is a read-only (download only) location.

Setup

Setting up such a profile is similar to the rest of profile types, with the main difference being that the user needs to manually provide the names and URLs for each file to be downloaded.

First create a new profile and give it a unique name.

Next, select the Source to be HTTP Download, and the destination to be your local or network drive.

Next, you can define some optional parameters related to this profile type.

• Number of download threads to use (too many will degrade performance): To improve performance you can download files in parts in parallel. For example, if you specify 5 threads, and a 50MB file is being downloaded, then it will download five 10MB parts in parallel. Once downloaded, the file is assembled locally from the parts.
• I use a proxy server: If you must use a proxy server to connect to external web servers then you must enable this option. Unless you're in a corporate environment it is unlikely that you use a proxy server.
• Hostname: This is the hostname of the proxy server, e.g. proxyserver.com. Variables can be used.
• Username: Your proxy login username. If you do not need to login to your proxy server then leave this blank. Variables can be used.
• Password: Your proxy login password. If you do not need to login to your proxy server then leave this blank.
• Port: The port number of the proxy server. This value varies depending upon the type of proxy server software used.

Once you've clicked Done you will be taken to the main profile configuration form where you will have to provide the Destination folder. This is the folder where the files will be downloaded into. You cannot change the HTTP Download folder (it is set to '\') as it has no meaning for this profile type.

Click the Choose sub-directories and files button to define the files and their corresponding URLs. The following form will appear. Click on the root folder (\) and select Add folder from the pop-up menu.

You will be prompted to name the folder. This is the name of the sub-folder we are going to download the files into the destination. In this example, we will name the folder 2BrightSparks. Click OK. The folder is now shown in the window.

Right-click on the folder and select Add file from the pop-up menu. You are now prompted to name the file. This is the name of the file we are going to download to. You can give it the same name as it has on the web server, or any other filename.

Next you will be asked for the URL to download the file from. In this example we'll use the URL https://www.2brightsparks.com/assets/pdf/SyncBackProV10.pdf

Click OK once you'e entered the URL. The new file now appears in the window.

You can now repeat the above steps to add more files and folders.

Here are some things to remember:

• To download a file to a folder (including the root folder) just right-click on it and select Add file (or press Ctrl-F).
• To create a new sub-folder right-click on the folder you want to create it in and select Add folder (or press Ctrl-D).
• If you want to remove a file or folder, select it and press the Delete key.
• To rename a file or folder, and change a file URL, select it and press the F2 key. You can also double-click on the name to just rename the file or folder without changing the URL.
• To change just the URL of a file, double-click the URL text.

Once you are done with the files and URLs definitions, click OK to save the changes and then click OK again to save the profile changes in general. Once saved, you can perform a simulation or do an actual profile run.

Points to Note

• The New Files and New Folders settings (e.g. Include new files) have no meaning when using HTTP download and are ignored.
• The download URL you enter is not validated and is taken as-is. If the URL is invalid or it does not point to a valid resource, then SyncBackPro will raise an error during profile simulation and execution.
• A file may be processed (altered) by the web server before being delivered. For example, if you try to download a HTML or PHP file, then the web server is unlikely to return the raw file (as it is on the server). Similarly, downloading a HTML file does not download any linked content such as images.
• Binary files, e.g. executable's, compressed files, etc. are usually delivered as-is without being changed. However, images may be altered by the web server or an intermediary cache service like Cloudflare.
• Files downloaded may be retrieved from a cache (local or remote), e.g. Cloudflare.
• Links are not followed, meaning if you download a HTML file, for example, then it only downloads that file and not any images etc. that it refers to.
• You can only download files, not upload files. You can specify URL parameters (query strings), but it will always be a GET call.
• If a file is created dynamically by the web server, then it is likely it will always be downloaded by SyncBackPro. This is because the server will probably not provide the file size and any last modification date and time will constantly change.