https://paltili.hatenablog.com/entry/2020/10/15/165702. There are two options for command line bulk downloading depending on the tools that you have available.
• WGET • cURL
Mar 23, 2017. Curl doesn't have an option to that (without also specifying the filename), but wget does. The directory can be relative or absolute. Also, the directory will automatically be created if it doesn't exist. Wget -P relative/dir '$url' wget -P /absolute/dir '$url'.
WGET Instructions - for command line in Mac and Unix/Linux
1. Configure your username and password for authentication using a .netrc file
Affinity photo free download windows.
echo 'machine urs.earthdata.nasa.gov login <uid> password <password>' >> ~/.netrc
where
<uid> is your Earthdata Login username and <password> is your Earthdata Login password. Do not include the brackets <> .
2. Use a WGET command to download your data. Example WGET command:
wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies --no-check-certificate --auth-no-challenge=on -r --reject 'index.html*' -np -e robots=off < insert complete data HTTPS URL >
WGET Instructions - for Windows
1. Create a text file to store the website cookies returned from the HTTPS server, called 'mycookies.txt'. Store this in the wget installation directory. Curl Default Download Directory Macro
2. Use a WGET command to download your data. Example WGET command:
Curl Default Download Directorywget --http-user=[USERNAME] --http-password=[PASSWORD] --load-cookies mycookies.txt --save-cookies mycookies.txt --keep-session-cookies --no-check-certificate--auth-no-challenge -r --reject 'index.html*' -np -e robots=off <insert complete HTTPS data URL>
WGET Tips
Mac and Unix/Linux users can also skip the step of storing the username and password in a netrc file by adding
--http-user=[USERNAME] --http-password=[PASSWORD] , or --http-user=[USERNAME] --ask-password .
The WGET examples provided in this article will download files from the specified directory to a directory on your machine. The directory on your machine will have the title of the HTTPS host. For NSIDC, this will either be 'n5eil01u.ecs.nsidc.org' or “daacdata.apps.nsidc.org”.
You can modify how the files are saved using the using the following WGET flags
: -nd (or --no-directories)-nH (or --no-host-directories)--cut-dirs=number (where the number is the number of directories to cut, but doesn't include the host directory name - you'd need to include -nH to remove that still)
The GNU WGET 1.18 Manual provides more details on WGET options.
CURL Instructions - for command line in Mac and Unix/Linux
Step 1 of the WGET instructions still applies, just use the cURL command instead of WGET. curl -b ~/.urs_cookies -c ~/.urs_cookies -L -n -O <insert complete HTTPS data URL>
The
-O option on the cURL command downloads the file to the current working directory on your computer.
Last updated: 26 June 2019
Comments are closed.
|
AuthorWrite something about yourself. No need to be fancy, just an overview. Archives
December 2020
Categories |