1. Approach
This documentation contains two scripts, one for Windows and one for Linux, which organise the downloading of files from the Internet and check their hash values (SHA-256) straight away to help ensure that the files have not been compromised.
It is a simple solution that combines downloading and checking. Everything is done in a single batch or shell script for Windows or Linux, using commands provided by the operating system.
The scripts themselves can be copied from here as plain text and saved as a batch file or shell script on your hard disk in a directory that is included in the operating system’s PATH.
Then you can handle all necessary downloads with a simple script file in the form of a "BOM" ("Software Bill of Materials" or just "download manifest").
The structure of the BOM file is explained using practical examples.
Some hints about safety and vulnerabilities are included here.
2. Really short instruction sheet for use without studying the background
The download-from-URL.bat for Windows is here:
Save it with the file name download-from-URL.bat in a directory included in the PATH,
for example, a suitable directory under C:\Programs.
The download-from-URL as Linux shell script is here:
Save it with the file name download-from-URL (without the timestamp and without .sh) in a directory included in the PATH,
for example /usr/local/bin or in an own directory such as /home/USER/myScripts/.
On Linux, make the file executable with chmod +x.
Look into these files only out of curiosity, or read the details in chapter The content of the download-from-URL.bat or chapter The content of the download-from-URL as shell script
Then you can copy a BOM file as plain text from Internet pages or write one yourself as batch file or Linux shell script (or even usable in a MinGW or such environment) in the following form (an example is explained in chapter The format of the BOM for download):
download-from-URL "$0" "%~0" ../ exit REM The following lines are not executed by the batch, but they are read as bom REM bom = software bill of materials # See https://www.vishia.org/SwEng/html/download-from-URL.html for organisation of this download GET-URL download-from-URL.bat https://vishia.org/SwEng/deploy/download-from-URL/download-from-URL-2026-11-02.bat 0c7e81d95857cfe0b62c7acba4fcc94100158808b1c269344c37f3460a61cb26 GET-URL download-from-URL https://vishia.org/SwEng/deploy/download-from-URL/download-from-URL-2026-11-02.sh 744321dcaa312901f29f9c27b419eb7e57f3fa2c40909656da96095b31d4cc6a # See https://www.vishia.org/Linux/html/mkSymbLink.html for explaination of this files GET-URL readlink.exe; https://www.vishia.org/Linux/deploy/readlink-2026-10-27.exe; 81c864e22c53f4aaf432834718aa5025f635e59503e4221c931f082a5829269a; GET-URL mkSymbLink.exe; https://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe; 588aadc021eb0cf215544278d2fa7763f0b145dac6d3bb3e298b5bdd5a514b70; GET-URL src/Linux-Windows-Tools-src-2026-10-27.zip; https://www.vishia.org/Linux/deploy/Linux-Windows-Tools-src-2026-10-27.zip; 77b1a6683be016651ad365f042a686fe78492046d396b999e10144ed27af9dc1;
This allows you to download these files and check their contents for unexpected modifications. In this example, there are two executables for Windows and a ZIP file containing their sources.
The download-from-URL script can always be used; the BOM file format is simple.
How can you obtain the SHA-256 hash? It is published on many web pages. Otherwise, you can calculate it yourself from the downloaded file.
3. Be careful when downloading executables and ZIP files from the Web
Downloading a file is often as simple as selecting ‘Save as’ for a link.
-
If you download a PDF or text file, this is usually fine.
-
If you download a ZIP file, you should check its contents before using them. If the ZIP contains only text files, PDFs or similar documents, the risk is usually lower, but not zero.
-
However, a ZIP file may have been compromised, and the unzip program may contain security vulnerabilities:
A ZIP archive can contain malicious executables or scripts. Normally, these are only unpacked, not executed. However, a specially crafted archive may exploit a security vulnerability in the unpacking software. Another possible attack is ''path traversal'', also known as Zip Slip, where manipulated file paths cause files to be written outside the intended destination directory.
-
If you download an executable, you should have a reason to trust the web page. However, consider that the web page may have been compromised recently, without the compromise having been detected yet, and malware may have been introduced. Another possibility is a "man-in-the-middle attacker" who changes the content.
That is why you should verify files whose contents cannot be inspected easily by checking a checksum, before saving them permanently on your hard disk.
A suitable checksum algorithm is the widely used SHA-256 hash.
Such verification is widely supported, and checksums are published on many web pages in a separate section. However, checking a checksum manually is inconvenient and can be error-prone.
That is why downloading and checking should be performed as one operation, without requiring manual steps.
The SHA-256 hash and the download location should be obtained independently, because an attacker who compromises a web repository may also replace the hash value stored alongside the file. Do not download the SHA-256 hash (possibly in the form of the BOM used here) and then download the files from the same location, if you want the hash to provide independent protection.
It is recommended to obtain the hash as plain text from an independent, trusted source. The whole BOM file used here can also be copied as plain text from another trusted location; using a different web server is preferable.
The tools used by the download-from-URL scripts are provided by the operating system
or are commonly available tools. Their integrity should be ensured by keeping the operating system
and its software sources trustworthy. You can also check the binary integrity of executables with this tool itself.
4. The format of the BOM for download
You need only a short text file, which is a BOM: a "Bill of Materials", or, more precisely for this purpose, a "download manifest". The BOM contains the files that need to be downloaded for a software application. The term "bill of materials" follows the established use of this term in hardware-related contexts.
One line in the BOM contains:
-
The file path/name to store on your destination. Restriction: Spaces are not admissible in this path. The destination path should be given either as an absolute path, or it is interpreted relative to the current working directory.
-
The URL to the download location.
-
The SHA-256 check sum (hash).
For simple use, the BOM file can also contain the commands that start the download script, so the BOM file can be an executable script by itself.
Furthermore any comment can be contained there.
A small example for such a BOM script file:
download-from-URL "$0" "%~0" ../ exit REM The following lines are not executed by the batch, but they are read as bom REM bom = software bill of materials # See https://www.vishia.org/SwEng/html/download-from-URL.html for organisation of this download GET-URL download-from-URL.bat https://vishia.org/SwEng/deploy/download-from-URL/download-from-URL-2026-11-02.bat 0c7e81d95857cfe0b62c7acba4fcc94100158808b1c269344c37f3460a61cb26 GET-URL download-from-URL https://vishia.org/SwEng/deploy/download-from-URL/download-from-URL-2026-11-02.sh 744321dcaa312901f29f9c27b419eb7e57f3fa2c40909656da96095b31d4cc6a # See https://www.vishia.org/Linux/html/mkSymbLink.html for explaination of this files GET-URL readlink.exe; https://www.vishia.org/Linux/deploy/readlink-2026-10-27.exe; 81c864e22c53f4aaf432834718aa5025f635e59503e4221c931f082a5829269a; GET-URL mkSymbLink.exe; https://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe; 588aadc021eb0cf215544278d2fa7763f0b145dac6d3bb3e298b5bdd5a514b70; GET-URL src/Linux-Windows-Tools-src-2026-10-27.zip; https://www.vishia.org/Linux/deploy/Linux-Windows-Tools-src-2026-10-27.zip; 77b1a6683be016651ad365f042a686fe78492046d396b999e10144ed27af9dc1;
The first line calls the download script on both Linux and Windows. Thus, only one file is needed for the BOM.
-
On Windows, it is a batch file named
download-from-URL.bat, which must be found through the PATH. -
On Linux, it is a shell script named
download-from-URL, which must also be found through the PATH.
Arguments:
-
The first and second arguments are alternative forms of the path to the BOM file, which is this file itself: one form for Linux and one for Windows. On the other operating system, the argument text is taken literally. The called script evaluates the first and second arguments. You can write whatever is necessary, but the arguments should start with
$and%respectively, so that each operating system can detect which argument belongs to it. For example, on Linux you can use a command substitution to refer to a file relative to this file, while evaluating it as an absolute path. On Windows, you can write something like"%~dp0\path"for the corresponding purpose. -
The 3rd and 4th arguments, here
../, specify the base path for storing the files from the BOM. You can provide two arguments for Linux and Windows expansion, or only one argument if both are the same. Here too, if both arguments are provided, they should start with$and%respectively, for the two operating systems. An environment variable can also be used, with the appropriate syntax for Linux and Windows. -
A relative path is often useful if the current working directory is the directory containing the files to be stored. Usually, the BOM file is stored there and started by double-clicking it.
In this example, the BOM file is intended to be saved in a download directory
inside a user’s directory that is part of the PATH, for example, C:\Programs\batch\download
or /home/myName/batch/download,
or, on Linux, in /usr/local/bin/download for system-wide use (this requires sudo or root privileges).
Then the files are stored in C:\Programs\batch, /home/myName/batch
or /usr/local/bin, respectively.
Each download directory can be kept, so that newer versions of the "software material" can be obtained when desired,
by changing the BOM file manually and carefully.
-
In general, use forward slashes in destination paths (a path/file combination is also possible in the BOM) for compatibility. The
download-from-URL.batscript also converts any backslashes in the path to forward slashes.exit
-
The second line,
exit, is not reached as a separate command after Windows execution because the batch file is invoked withoutcall. Control does not return to the calling script; execution continues in the called batch file until it ends. Otherwise,exitwould close the console by terminatingcmd.exe. -
On Linux, this
exitis necessary because the following lines must not be executed.
The following lines are either comments in several forms, for example here links to documentation what is downloaded here.
Lines starting with GET-URL are BOM entries, each describing one "software Material item":
In this example, three files are offered for download. The documentation explains what they are. This example is mainly useful on Windows; the files provide commands for creating symbolic links there. Symbolic links are supported by the Windows API (Application Programming Interface), but the available command-line support is not sufficient for all script use cases. You can also download these files on Linux.
The download also contains a ZIP file with the source code of these two executables.
It is provided as a service and is not required for normal use. If you comment out the corresponding GET-URL line,
you can prevent an unnecessary download.
5. Usages hints
5.1. The BOM file as argument, file not found
The path to the BOM file is expected either as first argument, or for Windows as second argument, if the first one starts with $.
Then the first is for Linux/UNIX, starting with a variable expansion. But this means, the BOM file itself should not start with $.
See example in The format of the BOM for download
and details argument parsing description in Check and set the BOM file and the destination folder for download and Check and set the BOM file and the destination folder for download
The path/to/bom-file can be only the file, then it is searched relative to the current directory.
Or it is a relative path, searched starting with the current directory,
or it is an absolute path which may be recommended in some circumstances.
It is possible to use an environment variable as start of this path, which should be given
in Linux with $ as first character, and in Windows as usual with `%`leftside.
The last one is important for detection of the destination directory for Linux.
The BOM file is tested whether it is existing and readable before usage:
BOM file not found: "c:/xPrograms/Batch\download/+download-vishiaLinux-Windows-Tools.bat"
That may occur on specific call of the script, not for the standard invocation shown in The format of the BOM for download. An error message occurs, here shown on a manipulated test call. The ERRORLEVEL is 5.
BOM file exists but not readable: "c:/xPrograms/Batch\download/+download-vishiaLinux-Windows-Tools.bat"
A second reason is, the file is existing but for example locked also for reading, hence a read error occurs. The ERRORLEVEL is 6
5.2. destination directory as argument, directory to store the downloaded file
The destination directory can be given as second to fourth argument, see The format of the BOM for download and details argument parsing description in Check and set the BOM file and the destination folder for download and Check and set the BOM file and the destination folder for download.
-
If the first argument starts with a
$, then it may be the third argument, because the second one is the BOM file. -
But if the third argument starts with
$, then it is for Linux/UNIX. Then use the 4th argument. -
If as result the destination would be
NOPAUSEbecause it is not intended, then it is set to empty.
The destination directory as argument has to be written with an ending slash.
If the destination directory is not given, then only the path/to/file is relevant.
If this is an absolute path (not recommended), it is taken.
Elsewhere the relative path starts from the current directory.
If the destination directory is given as argument, it should be existing on the given location. If this argument is a relative path (maybe recommended), then it is searched from the current directory.
If this destination directory is not existing, then the following error message is written:
destination directory does not exist: "dst\"
The destination directory has to be created manually.
Hint: If the path/to/file in the BOM is given as absolute path and a destination directory is given,
this situation is not well detected. Then the file is stored or searched before inside the destination directory
starting with the absolute path as relative one. This can be even sensible.
All sub directory levels given in the path/to/file are created automatically.
5.3. Existing files
The download-from-URL script checks whether the files already exist at the destination.
If they do, they are preserved. A new download is not forced.
However, these existing files are checked against the SHA-256 hashes in the BOM.
If the hash is correct, then only a short message, one line, is output:
file exists, hash Ok: path/name
This is useful when only missing files should be downloaded, for example, source files or additional files after changing the BOM, or when another BOM file is used and some files occur in both BOM manifest.
If a destination file already exists but has a different version (or is suspected of being compromised), a report is written to the console (stdout):
file exists with faulty hash: ../readlink; hash=cf4b19946f900285a043b29c35d4e48ee56cf75ca0b202cfe8499fd20f1a1e3b ^^^ERROR
The file is not removed because it
-
may be compromised; in that case, the user should remove it.
-
may be a new file for the BOM, created by the user, whose SHA-256 hash is now known.
-
may be an older version of the file listed in the BOM. The user may want to keep that older file and download only some newer files. This behaviour is intentional and leaves the decision to the user.
In all these cases, the user is responsible for taking the appropriate action:
-
remove a compromised file and investigate what happened.
-
use this file for a new BOM entry
-
comment the download BOM entry, or accept the situation on demand.
5.4. Download error
If a required file cannot be downloaded, a message in the following form is output:
Download ../xmkSymbLink.exe from https://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27777.exe ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404
ERROR: Download failed: ../xmkSymbLink.exe; https://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27777.exe
^^^ERROR
The first output comes from curl, the download command used here.
The ERROR output comes from the download-from-URL script itself.
What to do:
In this case, I have deliberately modified the URL, so it is invalid. You should check it, for example by testing whether the same URL is accessible in a web browser. If you copy the URL into the browser’s address bar, the browser should offer the file for download if it exists.
Of course, other network problems can occur, such as a disconnected RJ45 cable.
Some further examples:
Download ../xmkSymbLink.exe from https77://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe ... curl: (1) Protocol "https77" not supported ERROR: Download failed: ../xmkSymbLink.exe; https77://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe ^^^ERROR
Can you spot the mistake?
Download ../xmkSymbLink.exe from https://www77.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (60) schannel: SNI or certificate check failed: SEC_E_WRONG_PRINCIPAL (0x80090322) - Der Zielprinzipalname ist falsch.
More details here: https://curl.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.
ERROR: Download failed: ../xmkSymbLink.exe; https://www77.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe
^^^ERROR
Here, curl provides a longer error message because the server access has failed.
Download ../xmkSymbLink.exe from https://www.vishia.org77/Linux/deploy/mkSymbLink-2026-10-27.exe ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: www.vishia.org77
ERROR: Download failed: ../xmkSymbLink.exe; https://www.vishia.org77/Linux/deploy/mkSymbLink-2026-10-27.exe
^^^ERROR
Here, the server (host) was not found, as expected because the address is invalid.
Download ../xmkSymbLink.exe from https://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (6) Could not resolve host: www.vishia.org
ERROR: Download failed: ../xmkSymbLink.exe; https://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe
^^^ERROR
Here, the RJ45 cable is unplugged. Unfortunately, the error message is the same as for an invalid host. This is output from ''curl''.
Similar error situations can also occur with other network accesses.
5.5. Hash check failure for a downloaded file
This is the critical situation in which either the downloaded file is faulty or an incorrect hash is given on the Internet:
Download ../xmkSymbLink.exe from https://www.vishia.org/Linux/deploy/mkSymbLink-2026-10-27.exe ...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 102k 100 102k 0 0 597k 0 --:--:-- --:--:-- --:--:-- 603k
ERROR: SHA256 mismatch
Expected: 588gaadc021eb0cf215544278d2fa7763f0b145dac6d3bb3e298b5bdd5a514b70
Actual: 588aadc021eb0cf215544278d2fa7763f0b145dac6d3bb3e298b5bdd5a514b70
^^^ERROR
The calculated hash and the hash given in the BOM are reported so that you can compare them.
The hash itself cannot be checked manually, but perhaps a mistake was made when assembling the BOM.
Here, an extra character, g, was inserted into the hash. This may have been a typing mistake.
But if the hash consists of valid hexadecimal digits, you should investigate what happened to the downloaded file. Either the file has been compromised, or it is not the version expected by the given hash.
5.6. More usage possibilities, organise more downloads in a script
Although each BOM file has the format shown above, starting with download-from-URL followed by the BOM lines,
you can use a higher-level script to process several BOM files in a batch process.
This higher-level script can be written as a batch file, for example:
@echo off echo Download test >log.log call download-from-URL $x c:\Programs\Batch\download\+download-vishiaLinux-Windows-Tools.bat dst NOPAUSE >>log.log 2>&1 if ERRORLEVEL 1 echo ERROR %ERRORLEVEL% vishiaLinux-Windows-Tools.bat call download-from-URL $x D:\vishia\fbg\OFB_Presentation\tools\+download-vishiaOFB.bat dst NOPAUSE >>log.log 2>&1 if ERRORLEVEL 1 echo ERROR %ERRORLEVEL% +download-vishiaOFB pause exit /B
This is a practical test example. Two BOM files from different locations on the hard disk are used, for example, to test whether all downloadable files are available and have the expected SHA-256 hashes.
-
Execution should normally be silent. That is why the console output intended for manual use is redirected to a log file:
>>log.log 2>&1, for both stdout and stderr. The log file can be checked afterwards for the message^ERROR. -
However, a summary of errors for each BOM file may be useful to the tester. This is done by evaluating the
ERRORLEVELreturn value from each call ofdownload-from-URL.The higher-level script can abort the test when an error occurs, or implement other control logic.
5.7. Exit codes for script execution of download-from-URL
The script returns an exit code when it finishes.
On Windows, the calling batch file can evaluate this value using ERRORLEVEL.
On Linux, the calling shell can evaluate it using $?.
The exit codes can/should be used to decide how to continue with downloading processes. But the reaction is in the responsibility to the user.
The script uses the following return values:
Exit code 0..2: The script was running well, but if not 0, some situations have occurred.
-
0: No error; all required downloads are done and have the correct hash.
-
1: No errors occurred, but some files already existed and their SHA-256 hashes were correct.
-
This is usually not an error, but it may be interesting during a download test.
-
Hint: delete the existing files before such download test.
-
Exit code 3..4 for a maybe critical error but without download actions:
-
3: SHA-256 hash check failed for existing files:
-
Older files may be stored locally.
-
It is also possible that the files have been compromised locally.
-
All exit codes < 5 means, the execution of the script file is well done, but the situations explain above are occurred.
Exit codes 5..8: argument error, the script cannot work.
-
5: BOM file not found, presumed argument error
-
6: BOM file read error, it may be a more complicated file error, manual check necessary.
-
7: The given destination directory does not exist; the script cannot be executed.
-
should be created manually or it is an argument error.
-
Exit codes 9 ..12 Arguments of call are ok, but download not possible, generally or for some files, it is possible that the BOM file is wrong, check it:
-
9: Download error; something is wrong with the Internet connection or the URL, or the destination path is faulty and the file cannot be written.
-
Look at paths and internet connection, no trouble with attacked files.
-
-
10: Download errors for some files and failed SHA-256 checks for existing files.
-
Here both should be checked, but the error may be only a faulty BOM file.
-
Exit codes >= 13: Critical situation, check the reason:
-
13: SHA-256 hash check failed for downloaded files.
-
The files are not downloaded. You should check whether the files are compromised or whether there are new versions of files and your BOM should be updated.
-
-
14: Additional to download hash faulty, there are also some download errors.
-
this suggests changing the availability and correctness of the files on the URL source are given.
-
-
15: Both hash tests fails, on existing and on downloaded and then removed files.
-
Something is wrong with the files, the hash test or the BOM
-
-
16: All three error reasons are given, download fails, download done but faulty hash and also faulty hash on existing files.
6. The content of the download-from-URL.bat
This is the Windows implementation of the batch file, which downloads files and checks them against the BOM.
This batch file uses the cmd.exe batch language, which is available on Windows 10 and Windows 11.
Using PowerShell would be another approach. It is avoided here because cmd.exe is familiar to users of traditional batch files.
6.1. The whole file
This file is delivered in the https://vishia.org/SwEng/deploy/download-from-URL/download-from-URL-2026-11-02.bat
Save it with the file name download-from-URL.bat in a directory included in the PATH,
for example, a suitable directory under C:\Programs.
It is intended to universally applicable and can be used for all download requirements on Windows 10 and Windows 11 systems.
You can also copy the text from here and use it to save as content of download-from-URL.bat
It is licensed under the GNU Lesser General Public License (LGPL).
cls
@echo off
echo called: %0 %1 %2 %3
REM This file organises downloads from the web using the given BOM lines.
REM -- Arguments:
REM %1 is either "$1" in a combined Linux/Windows call, or see %2
REM %2 or even %1 is the path/to/file containing the BOM lines.
REM %2 or even %3 is the destination/base/path, if it is not "NOPAUSE"; it may be empty (not given)
REM if %3 starts with "$", it is for combined Linux/Windows,
REM then %4 is the destination path instead %3 or %2
REM The destination/base/path should end with "/".
REM Using forward slashes instead of backslashes is possible and recommended.
REM If none of %2, %3, %4 and %5 is "NOPAUSE", execution ends with a pause for manual use.
REM Calling variant for Linux and Windows:
REM download-from-URL "$0" "%0" "$DST/" "%DST%\" NOPAUSE
REM Calling variant possible for both:
REM download-from-URL "$0" "%0" "../dst/" NOPAUSE
REM Calling variant for Windows only:
REM download-from-URL "path\to\bom" "path\to/dst/" NOPAUSE
REM -- BOM
REM BOM lines start left aligned with "GET-URL ", followed by a space, and have the form:
REM GET-URL FILE; URL; SHA256-hash;
REM Other lines in this file are ignored. They can contain the 'download-from-URL' call or comments.
REM -- Usage:
REM This file is universally usable.
REM It is licensed under the GNU Lesser General Public License (LGPL) of the Free Software Foundation.
REM Written by Hartmut Schorrig, Germany. Version from 2026-11-02.
REM Note: This file was developed with assistance from ChatGPT.
REM ---
setlocal EnableExtensions EnableDelayedExpansion
set "EXISTFILEOK="
set "EXISTHASHFAIL="
set "DOWNLOADERR="
set "DOWNLOADHASHFAIL="
set "FILEBOM=%~1"
set "DST=%~2"
if "!FILEBOM:~0,1!" == "$" ( set "FILEBOM=%~2" & set "DST=%~3" )
if "!DST:~0,1!" == "$" set "DST=%~4"
if "%DST%" == "NOPAUSE" set "DST="
REM check FILEBOM
if not exist "!FILEBOM!" (
echo ERROR BOM file not found: "!FILEBOM!"
exit /b 5
)
type "!FILEBOM!" >nul 2>nul
if ERRORLEVEL 1 (
echo ERROR BOM file exists but not readable: "!FILEBOM!"
exit /b 6
)
if "!DST:~0,1!" == "$" set "DST=%~4"
if "%DST%" == "NOPAUSE" set "DST="
if not "%DST%" == "" if not exist "%DST%." (
echo destination directory does not exist: "%DST%"
if not "%~2" == "NOPAUSE" if not "%~3" == "NOPAUSE" if not "%~4" == "NOPAUSE" if not "%~5" == "NOPAUSE" pause
exit /b 7
)
for /f "usebackq eol=# delims=" %%L in ("!FILEBOM!") do (
set "LINE=%%L"
if "!LINE:~0,8!" == "GET-URL " (
for /f "tokens=2,3,4 delims=; " %%A in ("!LINE!") do (
set "FILE1=%DST%%%A"
set "FILE=!FILE1:\=/!"
set "URL=%%B"
set "HASH=%%C"
if exist "!FILE!" (
call :GetHash "!FILE!"
if /i "!ACTHASH!"=="!HASH!" (
echo file exists, hash Ok: !FILE!
set "EXISTFILEOK=!EXISTFILEOK! !FILE!;"
) else (
echo file exists with faulty hash: !FILE!; hash=!ACTHASH!
echo ^^^^^^ERROR
set "EXISTHASHFAIL=!EXISTHASHFAIL! !FILE!;"
)
) else (
echo Download !FILE! from !URL! ...
for %%F in ("!FILE!") do set "DIR=%%~dpF"
if not "!DIR!" == "" if not exist "!DIR!" mkdir "!DIR!"
curl.exe --fail --show-error --location --output "!FILE!" "!URL!"
if errorlevel 1 (
echo ERROR: Download failed: !FILE!; !URL!
del "!FILE!" 2>nul
echo ^^^^^^ERROR
set "DOWNLOADERR=!DOWNLOADERR! !FILE!;"
) else (
call :GetHash "!FILE!"
if /i "!ACTHASH!"=="!HASH!" (
echo ok
) else (
echo ERROR: SHA256 mismatch
echo Expected: !HASH!
echo Actual: !ACTHASH!
del "!FILE!" 2>nul
echo ^^^^^^ERROR
set "DOWNLOADHASHFAIL=!DOWNLOADHASHFAIL! !FILE!;"
)
)
)
)
)
)
echo == Downloaded Files with faulty hash removed: !DOWNLOADHASHFAIL!
echo == Files with download error: !DOWNLOADERR!
echo == Existing Files with faulty hash: !EXISTHASHFAIL!
echo == Files already existing, hash ok: !EXISTFILEOK!
if not "%~2" == "NOPAUSE" if not "%~3" == "NOPAUSE" if not "%~4" == "NOPAUSE" if not "%~5" == "NOPAUSE" pause
REM exit:
if not "!DOWNLOADHASHFAIL!" == "" (
if not "!EXISTHASHFAIL!" == "" (
if not "!DOWNLOADERR!" == "" exit /b 16
exit /b 15
)
if not "!DOWNLOADERR!" == "" exit /b 14
exit /b 13
)
if not "!EXISTHASHFAIL!" == "" (
if not "!DOWNLOADERR!" == "" exit /b 10
exit /b 3
)
if not "!DOWNLOADERR!" == "" exit /b 9
if not "!EXISTFILEOK!" == "" exit /b 1
endlocal
exit /b 0
REM Subroutine to get the HASH via certutil:
:GetHash
set "ACTHASH="
for /f "skip=1 tokens=*" %%H in ('certutil -hashfile "%~1" SHA256') do (
set "ACTHASH=%%H"
goto :HashFound
)
:HashFound
exit /b
The code should be understandable to anyone who wants to examine it. That’s why some details are explained:
6.2. Start: batch environment and variable expansion
setlocal EnableExtensions EnableDelayedExpansion
-
setlocalstarts a local environment for this batch file. Changes to environment variables remain local to this execution and are discarded when the batch file ends (or when a matchingendlocalis executed). See https://learn.microsoft.com/windows-server/administration/windows-commands/setlocal -
EnableExtensionsas argument ofsetLocalenables the command extensions ofcmd.exe, which provide additional syntax for commands such asifandfor. -
EnableDelayedExpansionas an argument ofsetlocalenables variable expansion using!NAME!. This is execution-time expansion: the variable is expanded when the command is executed, rather than when the entire parenthesised block is parsed. With%NAME%, the value may be expanded only once when the block is parsed, even if the command is executed repeatedly in a loop. Microsoft calls this feature 'delayed expansion'; here, 'execution-time expansion' or 'expanded on demand' is a more descriptive term.
set "EXISTFILEOK=" set "EXISTHASHFAIL=" set "DOWNLOADERR=" set "DOWNLOADHASHFAIL=" set "FILEBOM=%~1"
This are variables which are filled with file names on error, used for the exit code (ERRORLEVEL) and a report on end.
6.3. Check and set the BOM file and the destination folder for download
set "FILEBOM=%~1" set "DST=%~2" if "!FILEBOM:~0,1!" == "$" ( set "FILEBOM=%~2" & set "DST=%~3" ) if "!DST:~0,1!" == "$" set "DST=%~4" if "%DST%" == "NOPAUSE" set "DST="
These statements evaluate the calling arguments:
* By default, the 1st argument is the path to the BOM: FILEBOM.
* But if the call is prepared for both Linux and Windows, the first argument may start with $.
It is not expanded on Windows, so the second argument must contain the valid Windows path for FILEBOM
The same applies to the destination path for all files: it can be the 2nd argument if the first argument is not prepared for Linux (and is therefore the path to the BOM), or the 3rd argument in a combined Linux/Windows call, or the 4th argument.
-
But if this argument is
NOPAUSE, then no destination path is given; it is empty.
REM check FILEBOM if not exist "!FILEBOM!" ( echo BOM file not found: "!FILEBOM!" exit /b 5 ) type "!FILEBOM!" >nul 2>nul if ERRORLEVEL 1 ( echo BOM file exists but not readable: "!FILEBOM!" exit /b 6 )
To get an unique exit code for problems with the BOM-file, these statements are introduced: * The existence of the BOM-file is checked, on non existence the script should be finished with this ERRORLEVEL.
-
The problem that the file has a reading problems, for example because it is locked for writing and hence not accessible, it is read here only for test. If the
typedoes not return an ERRORLEVEL, it should be readable also in the next statements. -
However, this is not guaranteed, because in rarelay case the file can be removed or locked accidently in this moment after read check. But for the rarely case nothing is destroyed, only the ERRORLEVEL is not as expected.
Inferential, it may be better to read the file and store it completely in a variable, then parse it. This is not so far possible in this batch script language.
if not "%DST%" == "" if not exist "%DST%." ( echo %DST% does not exist; it should be a directory. if not "%~2" == "NOPAUSE" if not "%~3" == "NOPAUSE" if not "%~4" == "NOPAUSE" if not "%~5" == "NOPAUSE" pause exit /b 7 )
If the destination path is given, the batch script checks that it exists as a directory.
* Whereby it is tested exist "%DST%." The dot after %DST% points to the directory itself
because %DST% should have an slash (or possible backslash) on its end.
This slash is essential to build the path and hence tested here.
* But one exception exists. If the slash is not written on end, but an directory with the proper name with an trailing dot exists,
this test does not fail, but it is fail. This is a rarely expectable error, should be found manually.
The effect is, the files are stored with the name of the last part in %DST% as leading string of the filename
or a possible path/to/downloadfile. This effect should be able to detect.
If the destination path does not exist, this is an error in the calling arguments or in the prepared destination environment, and the script cannot work.
6.4. for loop for each line of the BOM file
The for statement provides a way to read the contents of a file.
for /f "usebackq eol=# delims=" %%L in ("!FILEBOM!") do (
set "LINE=%%L"
This reads the given BOM file line by line.
-
Syntax for
for: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/for?view=windows-server-2019 -
The
/foption makesforprocess the contents of a text file, a string, or the output of a command, line by line. The text of each line can be split into tokens according to specified delimiters. In this script,/freads the BOM file line by line. See [Microsoft Learn — for]: https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/for -
The
usebackqoption changes how quotation marks, single quotation marks and back quotation marks are interpreted in thefor /finput specification. In particular, it allows the filename containing spaces to be enclosed in double quotation marks. In this script, the outerfor /fusesusebackqbecause"!FILEBOM!"is the name of the BOM file. See also Microsoft Learn — for -
Because of
eol=, lines starting withare ignored. Empty lines are also ignored. -
%~1: The~removes surrounding quotation marks from the first argument%1. The resulting value is then enclosed in quotation marks. Without~, the argument could be enclosed in quotation marks twice. Arguments should always be written in this form. I had not known this despite more than 30 years of experience with batch files. ChatGPT pointed it out to me. :-)
6.5. Evaluate the content of each line in the BOM file, do download
if "!LINE:~0,8!" == "GET-URL " (
Only lines starting with GET-URL are used. The marker is uppercase and must start at the beginning of the line, followed by a space (not a tab).
!LINE:~0,8! extracts the first eight characters of the variable LINE (positions 0 to 7).
for /f "tokens=2,3,4 delims=; " %%A in ("!LINE!") do (
set "FILE1=%DST%%%A"
set "FILE=!FILE1:\=/!"
set "URL=%%B"
set "HASH=%%C"
This parses the line. Spaces and semicolons are delimiters.
The space after GET-URL is used as a delimiter, so FILE is the second token in the line, and so on.
-
The values for
FILEandURLin this BOM line must not contain spaces or semicolons. For Windows batch processing, avoid exclamation marks (!) in the BOM file path, destination path, or BOM entries, because!is used for execution-time variable expansion. -
If the file name or its path need to contain spaces (although this is not recommended), you should save it without spaces and rename it afterwards outside.
-
Spaces in URLs are usually replaced by
%20. -
The
FILEvariable is prepared as follows:-
DSTfrom the arguments is prepended. IfDSTis empty, nothing is added. The destination path argument must contain the necessary trailing slash. -
In the complete
FILEpath, backslashes are converted to forward slashes. This is done by!FILE1:\=/!(character replacement); see https://ss64.com/nt/syntax-replace.html
-
6.6. Check whether the file to download is already given, check its hash
if exist "!FILE!" (
call :GetHash "!FILE!"
if /i "!ACTHASH!"=="!HASH!" (
echo file exists, hash Ok: !FILE!
set "EXISTFILEOK=!EXISTFILEOK! !FILE!;"
) else (
echo file exists with faulty hash: !FILE!; hash=!ACTHASH!
echo ^^^^^^ERROR
set "EXISTHASHFAIL=!EXISTHASHFAIL! !FILE!;"
)
If the file already exists, it does not need to be downloaded. Its SHA-256 hash is checked against the expected hash.
-
GetHashis called as subroutine, see below.-
If it matches the value given in the BOM, a short message containing the file name is shown as confirmation.
-
If it does not match, the file is not removed. The calculated SHA-256 hash is shown, and the user is responsible for deciding what to do. The user may have downloaded the file at some earlier time.
-
-
One of the variables used to accumulate errors is updated; see below for its evaluation.
Hint: ^ is an escape character in batch files (similar to \ in strings in C/++ or Java).
Writing it twice, ^^, produces one caret character. Here it is used as an upward-pointing marker for the error message.
6.7. Download the file from URL and check hash
The ) else ( is related to the question if exist "!FILE!" ( above.
) else (
echo Download !FILE! from !URL! ...
for %%F in ("!FILE!") do set "DIR=%%~dpF"
if not "!DIR!" == "" if not exist "!DIR!" mkdir "!DIR!"
curl.exe --fail --show-error --location --output "!FILE!" "!URL!"
If the file is not found, it must be downloaded.
-
The directory for the download is created if it does not already exist. Note that the
DSTdirectory has already been checked for existence. This means that"!DIR!"can be nonexistent only if the download file name contains a directory path. The situation in which"!DIR!"is empty occurs ifDSTis not given and the file name contains no directory path. Thenif not exist "!DIR!"would be erroneous because the directory name is empty. -
The extraction of a directory name with the
%~dp…construct is possible for calling arguments, for example%~dp1, or for aforvariable, but not directly for a normal environment variable. That is why theforconstruction is necessary here.
The command curl.exe is a commonly used tool for transferring data with URLs. See
https://curl.se: The curl project
-
This tool performs the download.
curlis an established open-source tool on both Windows and Linux. It is included with current Windows versions and is commonly available on Linux systems. -
--locationsupports redirected URLs -
The downloaded file is stored at the path in
FILE, obtained by parsing the BOM line and prependingDSTif supplied. Without the SHA-256 hash check, the download would be complete at this point.
But:
if errorlevel 1 (
echo ERROR: Download failed: !FILE!; !URL!
del "!FILE!" 2>nul
echo ^^^^^^ERROR
set "DOWNLOADERR=!DOWNLOADERR! !FILE!;"
If the download fails, curl writes an error message.
There can be several reasons (Internet connection, invalid URL, and so on).
The error message is usually clear and readable.
In addition, the message "ERROR: …" with the file name is written here.
During use, the user should be clearly informed that an error has occurred.
A possibly partially downloaded file is removed to leave the destination in a clean state.
) else (
call :GetHash "!FILE!"
if /i "!ACTHASH!"=="!HASH!" (
echo ok
If the download succeeds, these statements are executed.
-
The
GetHashsubroutine is called from within the script itself, for better overview and organisation; see the next explanation. It fills the variableACTHASHused for comparison with the expected hash from the BOM. -
This variable is compared as text with the SHA-256 hash given in the BOM. Using the option
/iforces comparison without distinguishing uppercase from lowercase. This is necessary here because the hexadecimal digits in the SHA-256 code can be given in different cases.
) else (
echo ERROR: SHA256 mismatch
echo Expected: !HASH!
echo Actual: !ACTHASH!
del "!FILE!" 2>nul
echo ^^^^^^ERROR
set "DOWNLOADHASHFAIL=!DOWNLOADHASHFAIL! !FILE!;"
)
If the hash check fails, the expected hash is reported, and the downloaded file is removed. This final action ensures a clean state; otherwise, a possibly compromised file would remain on your hard disk.
-
This means that a download with an incorrect hash must not be used.
-
You should carefully determine whether the hash in the BOM is incorrect or the file has been compromised.
-
General remark about error handling: errors are reported so that they are visible during manual use. The batch script also returns an error code if an error occurs. This is helpful when the download script is not started and monitored manually, but is included in a higher-level download script. The return codes are implemented below.
6.8. Summary and exit
) ) echo == Downloaded Files with faulty hash: !DOWNLOADHASHFAIL! echo == Files with download error: !DOWNLOADERR! echo == Existing Files with faulty hash: !EXISTHASHFAIL! echo == Files already existing, hash ok: !EXISTFILEOK! if not "%~2" == "NOPAUSE" if not "%~3" == "NOPAUSE" if not "%~4" == "NOPAUSE" if not "%~5" == "NOPAUSE" pause
This marks the end of both for loops.
The accumulated contents of the error variables are shown, mainly for testing and overview.
If everything is OK, the first three lines are empty. The fourth line lists files that already existed
and passed the hash check; it is also empty if no such files existed.
The calling argument NOPAUSE can be given in the 2nd, 3rd, 4th or 5th position.
If it is not given, the batch execution pauses and displays Press any key to continue.
The NOPAUSE argument is useful if this batch file is called from another batch file.
REM exit:
if not "!DOWNLOADHASHFAIL!" == "" (
if not "!EXISTHASHFAIL!" == "" (
if not "!DOWNLOADERR!" == "" exit /b 16
exit /b 15
)
if not "!DOWNLOADERR!" == "" exit /b 14
exit /b 13
)
if not "!EXISTHASHFAIL!" == "" (
if not "!DOWNLOADERR!" == "" exit /b 10
exit /b 3
)
if not "!DOWNLOADERR!" == "" exit /b 9
if not "!EXISTFILEOK!" == "" exit /b 1
The return ERRORLEVEL is determined by the contents of the accumulated error variables.
See Exit codes for script execution of download-from-URL
-
exit /bis necessary becauseexitwithout/bterminates thecmd.exeprocess. -
The number after
exit /bbecomes the value ofERRORLEVEL. -
A condition such as
if errorlevel 5is true whenERRORLEVELis greater than or equal to 5.
REM Subroutine to get the HASH via certutil:
:GetHash
set "ACTHASH="
for /f "skip=1 tokens=*" %%H in ('certutil -hashfile "%~1" SHA256') do (
set "ACTHASH=%%H"
goto :HashFound
)
:HashFound
exit /b
The batch file continues with the subroutine, which is called using the label call :GetHash …arguments….
The batch language has a syntax reminiscent of the 1980s, but it still runs on current Windows systems.
-
The SHA-256 hash is calculated with the standard Windows command
certutil. This is comparable tosha256sumon Linux. Unfortunately,certutilis primarily designed for manual use and is not particularly convenient in a batch script. The SHA-256 hash appears on the second line of the output. -
The
forstatement processes the output ofcertutil …. Thecertutilcommand is written in(…)and is therefore executed to provide the input forfor. -
skip=1in theforstatement skips the first line; the next line contains the SHA-256 hash. -
goto :HashFoundleaves theforloop after the hash line has been read, similar tobreakin a C/++, Java or Python loop. -
exit /bends the subroutine. Because it was called withcall :GetHash, execution returns to the calling level and continues with the next line.exit /bends only this subroutine or batch-file context. By contrast,exitwithout/bterminates the entirecmd.exeprocess.
7. The content of the download-from-URL as shell script
This is the UNIX/Linux implementation of the script, which downloads files and checks them against the BOM.
The Bash implementation is somewhat simpler in some parts than the Windows implementation. In my view, the UNIX/Linux environment has generally maintained a more consistent and downward-compatible approach, with gradual enhancements rather than disruptive changes.
7.1. The whole file
This file is delivered in the https://vishia.org/SwEng/deploy/download-from-URL/download-from-URL-2026-11-02.sh
Save it with the file name download-from-URL (without the timestamp and without .sh) in a directory included in the PATH,
for example /usr/local/bin or in an own directory such as /home/USER/myScripts/.
On Linux, make the file executable with chmod +x.
It is intended to universally applicable on UNIX/Linux systems. You can also copy the text from here and use it. It is licensed under the GNU Lesser General Public License (LGPL).
#!/bin/bash
clear
echo "called: $0 $1 $2 $3 $4 $5"
## This file organises downloads from the web using the given BOM lines.
## -- Arguments:
## $1 is the path/to/file of the BOM file.
## $2 starts with "%" if it is an argument for Windows usage, or
## $2 or even $3 is the destination/base/path, if it is not "NOPAUSE".
## The destination/base/path should end with "/" or the last element
## is prefixed to all file names.
## $4 may start with "%" for the Windows destination path, not used here.
## All paths should use forward slashes, also in the Windows version.
## If none of $2, $3, $4 and $5 is "NOPAUSE", execution ends with
## "...press any key" for manual start.
##
## Calling variant for Linux and Windows:
## download-from-URL "$0" "%0" "$DST/" "%DST%\" NOPAUSE
##
## Calling variant possible for both:
## download-from-URL "$0" "%0" "../dst/" NOPAUSE
##
## Calling variant for Linux usage only:
## download-from-URL "path/to/bom" "path/to/dst/" NOPAUSE
##
## -- BOM
## BOM lines start with "GET-URL " and have the form:
## GET-URL FILE; URL; SHA256-hash;
## Other lines in this file are ignored. They can contain the
## 'download-from-URL' call or any comments.
##
## -- Usage:
## This file is universally usable.
## It is licensed under the terms of the LGPL
## (Lesser General Public License) of the Free Software Foundation.
## Written by Hartmut Schorrig, Germany.
## Version from 2026-11-02.
## Note: This file was developed with assistance from ChatGPT.
## ---
## Clear error and information variables used for accumulation.
EXISTFILEOK=""
EXISTHASHFAIL=""
DOWNLOADERR=""
DOWNLOADHASHFAIL=""
## Pause only if none of arguments 2 to 5 is "NOPAUSE".
## Read from the terminal, not from the BOM file.
pause_if_needed() {
if [[ "$1" != "NOPAUSE" && "$2" != "NOPAUSE" && "$3" != "NOPAUSE" && "$4" != "NOPAUSE" ]]; then
if [[ -r /dev/tty ]]; then
read -r -n 1 -s -p "...press any key" < /dev/tty || true
echo
fi
fi
}
## check FILEBOM
if [[ ! -f "$1" ]]; then
echo "ERROR: BOM file not found: $1"
exit 5
fi
if [[ ! -r "$1" ]]; then
echo "ERROR: BOM file exists but not readable: $1"
exit 6
fi
## Determine the destination directory.
DST="$2"
if [[ "${DST:0:1}" == "%" ]]; then DST="$3"; fi
if [[ "$DST" == "NOPAUSE" ]]; then DST=""; fi
## Check the destination directory.
if [[ -n "$DST" && ! -d "$DST" ]]; then
echo "$DST does not exist; it should be a directory."
pause_if_needed "$2" "$3" "$4" "$5"
exit 7
fi
## Read the BOM file line by line.
## IFS= prevents field splitting by read.
## -r prevents interpretation of backslashes.
while IFS= read -r LINE; do
## Remove a possible Windows CR at the end of the line.
LINE="${LINE%$'\r'}"
if [[ "${LINE:0:8}" == "GET-URL " ]]; then
echo "==== line = $LINE"
## Split the BOM line into its four fields.
IFS='; ' read -r CMD FILE URL HASH <<< "$LINE"
## The used FILE is relative to the destination directory.
FILE="${DST}${FILE}"
## File already exists: check its hash.
if [[ -f "$FILE" ]]; then
HASH_LINE=$(sha256sum -- "$FILE")
HASHACT="${HASH_LINE%% *}"
if [[ "$HASHACT" == "$HASH" ]]; then
echo "file exists, hash OK: $FILE"
EXISTFILEOK="${EXISTFILEOK}${FILE}; "
else
echo "file exists with faulty hash: $FILE hash=$HASHACT"
echo "^^^ERROR"
EXISTHASHFAIL="${EXISTHASHFAIL}${FILE}; "
fi
else
## File does not exist: download it.
echo "Download: $FILE from $URL ..."
DIR=$(dirname -- "$FILE")
if [[ ! -d "$DIR" ]]; then mkdir -p -- "$DIR"; fi
if ! curl --fail --show-error --location --output "$FILE" "$URL"; then
echo "ERROR: Download failed: $FILE $URL"
rm -f -- "$FILE"
echo "^^^ERROR"
DOWNLOADERR="${DOWNLOADERR}${FILE}; "
else
## Download succeeded: check its hash.
HASH_LINE=$(sha256sum -- "$FILE")
HASHACT="${HASH_LINE%% *}"
if [[ "$HASHACT" == "$HASH" ]]; then
echo "ok"
else
echo "ERROR: SHA-256 mismatch, file removed."
echo "Expected hash=$HASH"
echo "Actual hash=$HASHACT"
echo "^^^ERROR"
rm -f -- "$FILE"
DOWNLOADHASHFAIL="${DOWNLOADHASHFAIL}${FILE}; "
fi
fi
fi
fi
done < "$1"
## Summary.
echo "== Downloaded files with faulty hash, removed: $DOWNLOADHASHFAIL"
echo "== Files with download error: $DOWNLOADERR"
echo "== Existing files with faulty hash: $EXISTHASHFAIL"
echo "== Files already existing, hash OK: $EXISTFILEOK"
pause_if_needed "$2" "$3" "$4" "$5"
## Exit status.
if [[ -n "$DOWNLOADHASHFAIL" ]]; then
if [[ -n "$EXISTHASHFAIL" ]]; then
if [[ -n "$DOWNLOADERR" ]]; then exit 16; else exit 15; fi
fi
if [[ -n "$DOWNLOADERR" ]]; then exit 14; else exit 13; fi
fi
if [[ -n "$EXISTHASHFAIL" ]]; then
if [[ -n "$DOWNLOADERR" ]]; then exit 10; else exit 3; fi
fi
if [[ -n "$DOWNLOADERR" ]]; then exit 9; fi
if [[ -n "$EXISTFILEOK" ]]; then exit 1; fi
exit 0
The code should be understandable to anyone who wants to examine it. That is why some details are explained:
7.2. Start: Bash header and initialisation
#!/bin/bash clear echo "called: $0 $1 $2 $3 $4 $5" ## This file organises downloads from the web using the given BOM lines. ....
If the first line is #!/bin/bash, the Bash shell is used to execute the script.
It is the most widely used shell, with some useful extensions, and is available on modern UNIX and Linux systems.
-
Note: A more universal approach is to use only POSIX-compatible commands and syntax. POSIX was standardised in the 1980s to support the different UNIX systems available at that time, and it still plays an important role today. However, Bash and its extensions beyond POSIX are very familiar.
-
clearandecho called…are used to provide clear screen output during manual downloads. -
Note that variables in an
echoargument should be enclosed in"…"; otherwise, expansion of$…may cause word splitting and pathname expansion, or other unintended effects. When enclosed in"…",$variableor$1and similar parameters are expanded but remain part of a single argument. This is especially important when you do not know exactly what the variables contain.
## Clear accumulated error and information variables. EXISTFILEOK="" EXISTHASHFAIL="" DOWNLOADERR="" DOWNLOADHASHFAIL=""
This initialises the variables used to accumulate errors and build the return value.
-
Strictly speaking, defining these variables first is not necessary in a shell script.
-
However, these variables might already have values inherited from the calling environment. Clearing them is therefore necessary.
7.3. Subroutine pause_if_needed
## Pause only if none of arguments 2 to 5 is "NOPAUSE".
## Read from the terminal, not from the BOM file.
pause_if_needed() {
if [[ "$1" != "NOPAUSE" && "$2" != "NOPAUSE" && "$3" != "NOPAUSE" && "$4" != "NOPAUSE" ]]; then
if [[ -r /dev/tty ]]; then
read -r -n 1 -s -p "...press any key" < /dev/tty || true
echo
fi
fi
}
This is a small subroutine used twice.
* It is defined among the other statements, but executed only when called.
* The requirement is that the script waits for a key from the keyboard only
if none of the arguments has been set to NOPAUSE.
* Note that $1 is the first argument of this subroutine, not the first argument of the whole script.
This distinction can be confusing.
7.4. Check and set the BOM file and the destination folder for download
For Linux/UNIX usage the BOM file is always the first argument.
## check FILEBOM if [[ ! -f "$1" ]]; then echo "ERROR: BOM file not found: $1" exit 5 fi if [[ ! -r "$1" ]]; then echo "ERROR: BOM file exists but not readable: $1" exit 6 fi
To get an unique exit code for problems with the BOM-file, these statements are introduced:
* -f tests whether the given path refers to a regular file. On non existence the script is finished with this exit code 5.
-
After then,
-rtests whether the file is readable by the current user. If not, an exit code = 6 ends the script. Assumed now, it should be readable also in the next statements.
These are preliminary checks. The file could theoretically become inaccessible before the later while loop opens it.
The destination folder can be given as the second or third argument. For Windows, it can be the 4th argument, but that argument is not used here.
DST="$2"
if [[ "${DST:0:1}" == "%" ]]; then DST="$3"; fi
if [[ "$DST" == "NOPAUSE" ]]; then DST=""; fi
These statements obtain the possibly supplied $DST path, including the Windows form %DST%.
* A separate $FILEBOM variable is not necessary here because the BOM file path is always in $1.
By default, it is the 2nd argument. But if this argument starts with %,
it represents a Windows environment-variable form for the BOM file, and the 3rd argument may then be the destination path.
On Linux, the destination path is never the 4th argument; it may also be omitted.
-
${name:0:1}is Bash syntax for extracting a substring. Here it extracts one character from position 0, that is, the first character.
## Check the destination directory. if [[ -n "$DST" && ! -d "$DST" ]]; then echo "$DST does not exist; it should be a directory." pause_if_needed "$2" "$3" "$4" "$5" exit 11 fi
This construct checks whether a destination directory path has been given and whether it exists. If the check fails, the script is aborted; there is nothing more to do here.
-
In a Bash script,
if [[ … ]]is the modern Bash-specific form; this complete syntax is not specified by POSIX. See https://vishia.org/Linux/html/ShellScript.html#if-bash-extended -
First, it tests whether the shell variable
$DSTis not empty (that is, a destination was given), and then, using AND, whether that path exists as a directory. The corresponding POSIX test for the directory istest -d "$DST". The complete condition can be written with[ -n "$DST" ] && [ ! -d "$DST" ]. The[[construct replaces thetestcommand; it must be closed with]].
The form with [ … ] (one opening and one closing square bracket) is the older, more limited form.
7.5. While loop for each line of the BOM file
This subchapter explains the basics of reading lines from the BOM file
and how a while loop works.
## Read the BOM file line by line.
## IFS= prevents field splitting by read.
## -r prevents interpretation of backslashes.
while IFS= read -r LINE; do
## Remove a possible Windows CR at the end of the line.
LINE="${LINE%$'\r'}"
... ...
done < "$1"
This is the complete while loop, with an ellipsis representing its inner content.
-
The core command in the
whileloop isread -r LINE. Thereadcommand receives its input stream from the redirection at the end of the loop, afterdone:< "$1". This means that<"$1"supplies input to the commands in thewhileloop. -
read -r LINE < any-file.txtused outside of thewhileloop would mean, read one line from the input stream, which is provided here by the contents ofany-file.txt. Hence it reads the first line of this file. Thenreadstores the content in the shell variable$LINE. The effect of<any-file.txtis to open the file and provide its file handle as the input stream for the command, hereread. The file is closed after the loop finishes. -
When this
read … < input-streamis embedded in awhileloop, the input stream is opened (here, the file named by the 1st argument$1), and the loop body is executed betweenwhileanddoas long as the command returns true (that is, a return status of 0). The input stream is closed when the loop finishes.readreturns a non-zero status when no more lines are readable, which means that the end of the file has been reached. The current loop expects the last BOM line to end with a newline. This is a detailed explanation of the constructwhile …. read … do … done <inputstream. -
The
-roption disables the default behaviour ofreadin which the backslash acts as an escape character. With-r, the backslash is read as an ordinary character. Without-r, for example, a backslash at the end of a line escapes the newline and continues the input on the next line. Sequences such as\nand\tare not interpreted as newline and tab escape sequences byread. -
For details of the
readcommand, see https://man7.org/linux/man-pages/man1/read.1p.html.
The read command uses IFS. Its behaviour can also be affected by locale-related variables such as LANG and LC_ALL,
as well as NLSPATH and PS2; see the documentation.
-
IFSplays an important role forreadfor parsing the line. Here it is set to empty, because the whole line should be copied to theLINEvariable. -
The other variables do not control the field splitting performed here; they are relevant to locale handling, prompts or diagnostic messages.
LANGis usually set at system level. It specifies the default locale, for exampleLANG=en_GB.UTF-8, and can affect the language of diagnostic messages. -
LC_ALLcan influence the handling of characters and bytes, but it does not perform encoding conversion (for example, between UTF-8 and ISO-8859-1). For more details aboutLC_*, see https://www.gnu.org/software/gettext/manual/html_node/Locale-Environment-Variables.html. Here, 'locale' does not mean local variables; it refers to "localisation", including language and character-handling settings.LC_ALLis not set explicitly here because it is not needed for the line parsing performed by this script. -
Note that setting a variable before a command on the same line applies that value only to the immediately following command, here
read. Alternatively,IFScould be set in a separate statement before thewhileloop. -
LINE="${LINE%$'\r'}": This removes a carriage-return character (usually written as\r) from the end of the line. Otherwise, it would remain if the file had been written on Windows with DOS-style line endings. Note: Windows text files can also use a single line feed (Unix-style line ending) and can be processed. Saving files with Unix-style line endings is recommended.
General remark about the encoding of the file read by this script
UTF-8 is preferred; alternatively, use only ASCII characters (7-bit).
ASCII-only URLs are possible because URLs provide ways to represent non-ASCII characters,
for example, by percent-encoding the UTF-8 bytes of the German ä as %C3%A4. See, for example, https://seoeffekt.de/umlaute-in-seo/,
This site recommends avoiding umlauts in URLs.
A space in a URL is represented by %20. Spaces in file names should be avoided; they are not supported here.
For downloaded file names and paths, other special characters should also be avoided.
If special characters are necessary, UTF-8 encoding is now the preferred choice.
7.6. Evaluate the content of each line in the BOM file, do download
if [[ "${LINE:0:8}" == "GET-URL " ]]; then
echo "==== line = $LINE"
This tests LINE and processes only BOM lines starting with `GET-URL `, including the trailing space (not a tab character).
This is the marker for a BOM line.
-
The expression
${LINE:0:8}extracts the first eight characters of the line. The expected characters are all 7-bit ASCII, so this works with ASCII-compatible encodings. UTF-16 is not supported by this test.
## Split the BOM line into its four fields.
IFS='; ' read -r CMD FILE URL HASH <<< "$LINE"
The read command is used again, but this time to parse a single line:
-
the
<<<operator is an enhancement of the bash shell (not standardised in POSIX). It provides the contents of the given string as 'stdin' (standard input) for the command, hereread. That’s whyreadparses the$LINE. -
The
IFSvariable is used by thereadcommand. Here it is set to `; `, so both spaces and semicolons are field separators. Note that consecutive IFS whitespace characters do not create empty fields. The semicolons separate the fields; spaces around them are also treated as separators. -
The fields are assigned to the four named variables
CMD,FILE,URLandHASH. As expected,CMDreceivesGET-URL.
## The used FILE is relative to the destination directory.
FILE="${DST}${FILE}"
The FILE variable is prefixed with DST. If DST is empty, FILE remains unchanged.
7.7. Check whether the file to download is already given, check its hash
## File already exists: check its hash.
if [[ -f "$FILE" ]]; then
HASH_LINE=$(sha256sum -- "$FILE")
HASHACT="${HASH_LINE%% *}"
The existence of a regular file is tested with test -f "$FILE", here used in the if construct.
If test returns 0 (success), the then branch is executed:
Writing VAR=$(command args) executes the command in parentheses and assigns its standard output to the variable.
Here, the sha256sum command is called. -- marks the end of options, so the following file name is not interpreted as an option.
This is important because a file name in $FILE might otherwise be interpreted as an option if it begins with a hyphen.
This prevents sha256sum from being invoked with unintended options.
The construct "${HASH_LINE%% *}" is a Bash-internal operation on the contents of the given variable.
It does not need to call an extra command. This is useful for typical string-processing situations.
-
Here, all content starting at the first space is removed. The pattern ` ` means a space followed by any number of characters (the
wildcard). For this pattern,%%removes everything from the first space to the end of the string. -
See the Bash Reference Manual: Shell Parameter Expansion (https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html).
-
The hash value is the first item in the output of
sha256sum, followed by whitespace and the file name. Hence, the hash is extracted and the file name is removed.
if [[ "$HASHACT" == "$HASH" ]]; then
echo "file exists, hash OK: $FILE"
EXISTFILEOK="${EXISTFILEOK}${FILE}; "
else
echo "file exists with faulty hash: $FILE hash=$HASHACT"
echo "^^^ERROR"
EXISTHASHFAIL="${EXISTHASHFAIL}${FILE}; "
fi
If the hash is incorrect, the file that already existed is not removed. It may be another version of a valid file. However, the mismatch is reported.
Correct existing files are also recorded in the EXISTFILEOK accumulation variable, to build the appropriate return value (exit status).
7.8. Download the file from URL and check hash
If the file does not exist, it is downloaded and its hash is checked:
else
## File does not exist: download it.
echo "Download: $FILE from $URL ..."
The else belongs to if [[ -f "$FILE" ]]; then; this branch is executed when the file needs to be downloaded.
The echo is useful because the following curl command does not report the destination file name by itself.
DIR=$(dirname -- "$FILE")
if [[ ! -d "$DIR" ]]; then mkdir -p -- "$DIR"; fi
Before downloading, ensure that the directory exists if a directory is specified.
The general $DST directory must be created manually. This helps prevent files from being written to unexpected locations
if an argument is incorrect. However, $DIR, as part of the path/to/file, should preferably be specified as a relative path.
This path is part of the download specification (the BOM line), so it is created if it does not exist.
In my use cases, source files are often downloaded beside the executables in ZIP format,
but into a separate directory such as srcTools.
if ! curl --fail --show-error --location --output "$FILE" "$URL"; then
echo "ERROR: Download failed: $FILE $URL"
rm -f -- "$FILE"
echo "^^^ERROR"
DOWNLOADERR="${DOWNLOADERR}${FILE}; "
The download itself is performed with curl, a common open-source tool on both Windows and Linux.
The options are the same as described in the Download the file from URL and check hash.
Here, the curl command is used directly as the condition of the if, instead of a test such as [[ … ]].
The ! negates the command’s exit status, so the condition is true if curl returns a non-zero status.
Thus, if curl returns an error (a non-zero exit status), the if … then branch is executed.
-
curlitself writes its error message to stderr. In addition, the script writes a clear message usingecho. -
If the file is only partially downloaded, this remainder is removed because the incomplete file is unusable and would be confusing.
-
The cumulative variable
DOWNLOADERRis filled for error return later.
else
## Download succeeded: check its hash.
HASH_LINE=$(sha256sum -- "$FILE")
HASHACT="${HASH_LINE%% *}"
If curl succeeds, the else branch after if ! curl … is executed.
The hash of the downloaded file is then checked in the same way as for an existing file.
if [[ "$HASHACT" == "$HASH" ]]; then
echo "ok"
else
echo "ERROR: SHA-256 mismatch, file removed."
echo "Expected hash=$HASH"
echo "Actual hash=$HASHACT"
echo "^^^ERROR"
rm -f -- "$FILE"
DOWNLOADHASHFAIL="${DOWNLOADHASHFAIL}${FILE}; "
fi
Here, if the hash is incorrect, the file is then removed. A hash mismatch may indicate a malicious or otherwise unexpected file, which should not be kept on the hard disk. The calculated hash of this file is also reported. It is possible that the file itself is correct and only the hash value in the BOM is wrong. This is the user’s responsibility; the matter must be clarified, and the download should be attempted again.
7.9. Summary and exit
These statements are executed after the
while ... done < "$1"
## Summary. echo "== Downloaded files with faulty hash, removed: $DOWNLOADHASHFAIL" echo "== Files with download error: $DOWNLOADERR" echo "== Existing files with faulty hash: $EXISTHASHFAIL" echo "== Files already existing, hash OK: $EXISTFILEOK"
This is not merely a debugging aid; it provides an overview of a possibly long BOM file. If the script is called with stdout and stderr redirected to a log file, these messages can be found in the log.
pause_if_needed "$2" "$3" "$4" "$5"
The pause (waiting for press any key) is used for manual invocation of the download.
A longer output is then written to the console, and the display pauses.
If errors occur, they can be detected visually. You can then copy the console output
or inspect the files.
If all files have already been downloaded and are OK, only one line per file is written to the console.
If the script is started from another script, the NOPAUSE argument should be used to prevent the …press any key prompt.
The output can also be redirected to a log file.
## Exit status.
if [[ -n "$DOWNLOADHASHFAIL" ]]; then
if [[ -n "$EXISTHASHFAIL" ]]; then
if [[ -n "$DOWNLOADERR" ]]; then exit 16; else exit 15; fi
fi
if [[ -n "$DOWNLOADERR" ]]; then exit 14; else exit 13; fi
fi
if [[ -n "$EXISTHASHFAIL" ]]; then
if [[ -n "$DOWNLOADERR" ]]; then exit 10; else exit 3; fi
fi
if [[ -n "$DOWNLOADERR" ]]; then exit 9; fi
if [[ -n "$EXISTFILEOK" ]]; then exit 1; fi
exit 0
Especially when this script is called from another script, its exit status can be evaluated so that the calling script can abort if a meaningful error occurs. See chapter Exit codes for script execution of download-from-URL
8. A final note: This documentation and the scripts were developed with the help of ChatGPT
A little bit of history: Downloading files and automatically checking them was something I had already implemented about ten years ago.
I realised that wget, commonly used on Linux, was not available on Windows, and my focus at that time was on Windows.
That is why I developed a Java solution to do the job. It used an MD5 checksum to verify the downloaded files.
But there was a disadvantage: The program itself was a JAR file of about 100 kByte, which might be considered unsafe. It was nevertheless checked against its own hash.
A second disadvantage was that Java is not available by default on every system. This meant that the approach was suitable if Java was already needed for other purposes, but not for small, independent tools.
Then came another challenge: I needed small tools under Windows to create symbolic links compatible with Linux, without assuming that a MinGW/MSYS environment was available. Only small scripts and tools for Windows-only usage should be necessary, working in a similar way to their Linux counterparts.
This challenge was addressed (although the solution is not yet fully finished; see https://www.vishia.org/Linux/html/mkSymbLink.html). For this challenge, too, I asked ChatGPT about possible approaches and the best way to implement them.
Once these tools were usable, although not yet completely finished, another challenge arose: how to offer them to other people.
-
Simply offering an executable and asking people to trust it is not a reasonable approach.
-
Using my Java downloader with its MD5 check was another possibility. But MD5 is now outdated, so it should first be updated to SHA-256. And the other disadvantages mentioned above would still remain.
-
Was there a new approach? I asked ChatGPT, and the answer was yes:
curlis available for both Windows and Linux,sha256sumcan be used on Linux, andcertutilon Windows.
But how should these tools be used? The official documentation is sometimes not particularly easy to understand. I experimented with the suggestions from ChatGPT, and the first scripts were created.
In the past, I had often worked with DOS batch files under Windows, later with shell scripts under Windows using MinGW and Cygwin,
and of course with shell scripts under Linux.
But I had never received any formal further training in these scripting environments, nor had I used all their possibilities and specifics.
For example, the for loop in DOS/Windows had long been something of a mystery to me.
I prefer programming in Java, or perhaps in C/++, because the concepts are more obvious to me.
-
Ask ChatGPT for details.
Then I wrote this short documentation for the scripts, because I wanted them to be usable by other people, too.
First, my English could certainly be improved. Second, I needed a sparring partner to read the documentation, discuss the concepts and the state of the art, and help identify small mistakes. ChatGPT became that partner.
And last but not least, ChatGPT also found some programming mistakes in the scripts themselves
(for example, missing quotation marks when using "!VAR!", and other errors).
Of course, it also helped to improve the English.
I think ChatGPT is a helpful tool. It can draw on a broad range of knowledge, carry out research, and help compare and assess different approaches. It is not a human intelligence, but it can bring together and make useful connections between knowledge and ideas developed by humankind.
ChatGPT also makes mistakes. Sometimes it has difficulties understanding human-written documentation, needs time to find the best source, or draws conclusions too quickly from incomplete input.
That is why a partnership is necessary. ChatGPT makes a proposal, I test it and give feedback, and ChatGPT may respond: "Yes, sorry, we should try the other approach."
This is a normal working process of cooperation.
And one last remark: I could also have asked real people instead of ChatGPT. But when I am working on a problem, the people whose help would be useful are often not available at that particular moment. It is a question of timing.
This documentation should certainly be read and reviewed by real people, and I hope to receive their feedback. But I expect that this may take three or four weeks, perhaps even a month.
Dr. Hartmut Schorrig Doctorate in Computer Science, 1986