Active3 months ago
Requests is a really nice library. I'd like to use it for download big files (>1GB).The problem is it's not possible to keep whole file in memory I need to read it in chunks. And this is a problem with the following code
- Azure Storage Blob Download Speed Test Test download speed from Azure Storage Service around the world.
- Uploadfiles.io is a completely free file sharing service, upload files without limits, securely & anonymously. The simplest way to send large files, instantly. No signup required, just drag, drop & share.
- The fast way to send large files, no registration required! Share files up to 50GB for free via link or e-mail. Secure file transfer.
By some reason it doesn't work this way. It still loads response into memory before save it to a file.
The best way to send and share big files. Send large files with MailBigFile. Upload up to 4GB per file. We transfer your files for Free! TransferNow is available on all your devices: laptop, tablet, and smartphone. Send and share large files quickly in just a few clicks, and get a download link that will be sent directly to your contacts, or easily get a custom link and share it however you'd like on forums and social media. Download large file in python with requests. Ask Question Asked 6 years, 4 months ago. Active 3 months ago. Viewed 252k times 331. Requests is a really nice library. I'd like to use it for download big files (1GB). The problem is it's not possible to keep whole file in memory I need to read it in chunks.
UPDATE
If you need a small client (Python 2.x /3.x) which can download big files from FTP, you can find it here. It supports multithreading & reconnects (it does monitor connections) also it tunes socket params for the download task.
Acumenus27.4k77 gold badges7777 silver badges7979 bronze badges
Roman PodlinovRoman Podlinov12.6k77 gold badges3535 silver badges5252 bronze badges
4 Answers
With the following streaming code, the Python memory usage is restricted regardless of the size of the downloaded file:
Note that the number of bytes returned using
iter_content
is not exactly the chunk_size
; it's expected to be a random number that is often far bigger, and is expected to be different in every iteration.See http://docs.python-requests.org/en/latest/user/advanced/#body-content-workflow for further reference.
Community♦
Roman PodlinovRoman Podlinov12.6k77 gold badges3535 silver badges5252 bronze badges
It's much easier if you use
Response.raw
and shutil.copyfileobj()
:This streams the file to disk without using excessive memory, and the code is simple.
vog12.2k66 gold badges4141 silver badges6060 bronze badges
![Large file download fails Large file download fails](/uploads/1/2/4/8/124820520/771743311.jpg)
165k1919 gold badges198198 silver badges320320 bronze badges
Not exactly what OP was asking, but.. it's ridiculously easy to do that with
urllib
:Or this way, if you want to save it to a temporary file:
I watched the process:
And I saw the file growing, but memory usage stayed at 17 MB. Am I missing something?
x-yurix-yuri6,07155 gold badges5151 silver badges9292 bronze badges
Your chunk size could be too large, have you tried dropping that - maybe 1024 bytes at a time? (also, you could use
with
to tidy up the syntax)Incidentally, how are you deducing that the response has been loaded into memory?
It sounds as if python isn't flushing the data to file, from other SO questions you could try
f.flush()
and os.fsync()
to force the file write and free memory;Community♦
danodonovandanodonovan14.2k66 gold badges5555 silver badges6767 bronze badges
Not the answer you're looking for? Browse other questions tagged pythondownloadstreampython-requests or ask your own question.
by Martin Brinkmann on August 03, 2014 in Internet - Last Update: December 08, 2014 - 49 comments
File downloads are just a matter of clicking on a link and waiting for the download to finish in most cases. There are however situations where downloads are interrupted so that you end up with a broken file on your system that you cannot do anything with.
Several users who tried to download the excellent Tech Toolkit I reviewed yesterday for example have reported that downloads of the 1.6 Gigabyte archive would fail whenever they tried to download it.
It can be quite frustrating if that happens regularly, especially if your computer's Internet connection is not super fast. If you download with 50 Mbit or more, you may not mind if you have to repeat the download, but if you are on a slower connection, you may very well do.
As far as solutions are concerned, there is only one that is suitable and that is to use download managers. The main benefit of download managers is that they support resume.
While the server the file is hosted on needs to support it as well, it ensures that the download will continue after it was interrupted previously.
Instead of having to download the large file over and over again from the beginning, downloads would restart from where the previous download stopped (with a little overhead).
Download managers may support additional features such as download acceleration, scheduling, or grabbing of media.
Free download managers
The following selection lists several download managers that you can use to download files of any size to your local system. Some integrate into web browsers while others may need to be started manually instead to pick up the downloads.
Requirements
- A free version of the application needs to be provided.
- The last update cannot date back more than two years.
- It needs to support resume.
- Needs to support Windows.
Free Download Manager
Available as a full and lite version (lite ships without Bittorrent support, video conversion plugin and languages), it offers to add plugins to Firefox or Chrome. These plugins are not required but they make things easier for users of supported browsers.
![Dropbox Dropbox](/uploads/1/2/4/8/124820520/219342507.png)
Tip: A click on File > Create portable version allows you to create a fully portable version of the download manager.
A right-click on a file and the selection of 'download with Free Download Manager' sends the download to the application where it will be processed.
The program supports a variety of features such as scheduling downloads, adding it to groups or authentication.
The download manager splits files into segments which it downloads separately from each other to improve the download speed. In addition to that, it supports resume so that interrupted downloads can be restarted where they stopped.
FlashGet
FlashGet does not ship with browser extensions but it monitors the Windows clipboard for file links and will pick those up automatically so that it is easy to add downloads to the application.
It highlights the size of the file that will be downloaded to the local system, and supports multiple download threads, authentication and options to categorize downloads.
The download manager supports resume so that broken downloads are a thing of the past, provided that the server is also supporting resume. R5 antenna manual.
EagleGet
The EagleGet download manager is available as a portable version and installer. The program monitors the clipboard automatically but can also be integrated into web browsers directly via its universal capture mode.
That's however not necessary to add downloads to it. Since it monitors the clipboard, all you have to do is copy links pointing to files to the clipboard so that they are picked up automatically by the software.
EagleGet ships with a truckload of features such as download scheduling, batch downloads, download acceleration using threading, a speed limiter or options to resume broken downloads.
uGet
The Linux download manager is also available as a Windows build. It supports clipboard monitoring to pick up files automatically if they have a matching file extension.
The download dialog that opens prior to that enables you to make modifications to the process. Here you can add authentication information, select the number of retries and the delay between retries, change the number of connections per server, or limit the download speed.
We carry software that are essential for students – including SPSS, Microsoft, and more – for up to 90% off retail price! Is it true I can download free software? Yes, it is true! We work with select schools and departments to provide no-cost software tools to students and faculty. Free software for college students.
GetGo Download Manager
The download manager ships with a browser built-in which makes it feel bloated, especially if you don't require that. It does monitor clipboard events though and will pick up downloads automatically.
It supports the usual feature set, from multi-threaded downloads to speed things up to authentication and the pausing/resuming of downloads.
File Hoster downloads
The programs listed in this category have been designed specifically for so-called file hosting services. They download files from sites such as Mediafire or Mega.
Note: Programs listed in this category may contain offers (adware) when you install them. It is highly recommended to pay attention to the installation dialog and select custom when possible to stay in control.
Free Rapid Downloader - The program requires Java to run. It supports more than 700 sites according to the feature list on the developer website. In addition to that, it ships with captcha recognition, proxy lists, and other comfortable features such as automatic shutdown of the computer or support for streaming videos.
JDownloader - The program supports hundreds of file hosting services but requires Java to run. It monitors the clipboard and will add downloads automatically to its queue if they are hosted on a supported server. The cross-platform program supports many extra features such as support for premium accounts, browser integration, OCR modules or the automatic extraction of password protected archives.
MiPony - The program supports hundreds of file hosting services and extra features just like JDownloader does. This includes support for several captcha plugins to automate the process of filling those out, web remote control options, and automatic database updates to avoid issues with select hosts. Note: may be blocked as malicious by some browsers / antivirus programs, therefore, no link.
PyLoad - The program does not support as many hosters as JDownloader or MiPony, but it may make up for it in other ways. It has been designed with low hardware requirements in mind, and while it makes sure of that, it does not sacrifice core functionality for it.
With that said, it is difficult to set up as you need to run a configuration script first (on the command line) and run a core program first before you can connect clients to it.
Which one is best?
Program Name | Portable | Resume | Monitoring | Remote Access | Schedule | Other features |
EagleGet | yes | yes | yes | no | yes | Traffic limits, download acceleration, video downloader, silent mode |
FlashGet | no | yes | yes | no | no | Traffic limits,download acceleration Bittorrent support, Emule support |
Free Download Manager | yes | yes | yes | yes | yes | Traffic limits, partial zip downloads, HTML spider, download acceleration, Bittorrent support |
GetGo Download Manager | no | yes | yes | no | yes | Traffic limits, download acceleration, web video downloads |
uGet | yes | yes | yes | no | yes | Traffic limits, download acceleration, command line version, |
There is no definitive answer to that question. It depends on what you require more than anything else. Do you want integration into web browsers or is clipboard monitoring or manual pasting of download links sufficient?
Do you require features such as support for authentication or proxy servers, scheduling or support for protocols such as Bittorrent or ftp?
Large File Download Speed
With all that said, Free Download Manager gets our recommendation.
Commercial Alternative: Internet Download Manager
How to download large files (best download managers)
Description
Our download manager overview provides you with a list of programs that you can use to download files from the Internet.
Large File Downloader For Android
Author