[paramiko] SFTP performance for large files

[Robey Pointer]

On 14 Dec 2006, at 14:07, Bill Zingler wrote:

> Hello Wouter,
>
> I have attached my test scripts which I'm using for testing secure  
> (scp, sftp) and non-secure (ftp) file transfer clients on both  
> Linux and Windows to a Linux server.  All of my testing is being  
> run over a dedicated Gigabit Ethernet between the client and server.
>
> I find that paramiko will run slower by a factor of 2 or 3 as  
> compared to scp on Linux and pscp on Windows for files that are  
> less then 1 MB.  This is quite reasonable for pure python  
> implementation compared to a C or C++ implementation that scp and  
> pscp are.  But as I tested file sizes in the 100 MB and 1 GB range,  
> I find that paramiko will run slower by a factor of 20.  I'm sure  
> that there is some room for optimization.
>
> The testing is controlled by the .cfg file, so you can tailor it to  
> change file sizes and repetitions.  The script requires the  
> ConfigObj module which you can get from the Cheese Shop.  Other  
> than that, the remaining dependencies should be standard ones.
>
> I would really like to use to use paramiko in my application and  
> would if only small files were needed to be transferred.

I tried this on my local (home) network, from a macbook pro (laptop)  
to a Linux box, but got different results: the transfer rate slightly  
improves as the file size gets larger.  Actually this is what I would  
expect, since the initial handshake is included in the timing, and  
for smaller files, that will be significiant.

-----
paramiko took 10.501155 seconds to write a 1MB file 10 times
    with a transfer rate of 0.952276 MB and avg xfer time of 1.050115  
sec

paramiko took 76.212235 seconds to write a 10MB file 10 times
    with a transfer rate of 1.312125 MB and avg xfer time of 7.621223  
sec

paramiko took 152.376369 seconds to write a 20MB file 10 times
    with a transfer rate of 1.312539 MB and avg xfer time of  
15.237637 sec

paramiko took 368.105170 seconds to write a 50MB file 10 times
    with a transfer rate of 1.358307 MB and avg xfer time of  
36.810517 sec

paramiko took 764.182173 seconds to write a 100MB file 10 times
    with a transfer rate of 1.308588 MB and avg xfer time of  
76.418217 sec

paramiko took 1466.026837 seconds to write a 200MB file 10 times
    with a transfer rate of 1.364232 MB and avg xfer time of  
146.602684 sec
-----

One thing to note: I'm using a modified version of pyCrypto, from here:

	https://launchpad.net/people/robey/+branch/pycrypto/devel

which fixes a performance problem in the crypto layer.  (I should  
probably do a new release soon.)  I don't think that would affect  
large files disproportionately, though, it just makes all transfers  
slower -- hence the 2-3x slowdown you're seeing.

What kind of network were you trying over?  If it's a high-latency  
link, paramiko might be filling up memory with pipelined data, and  
maybe we should limit that.  Currently it will dump as much data as  
it can.

Attached is the test script I used, in case there's something I left  
out.  It's basically a truncated version of the one you posted, but  
only using paramiko.  If you have lsprof installed, and you change  
'xfer' to 'xfer_profiled' in the script, it will profile the put()  
call and generate call traces, which might add useful info.

robey


-------------- next part --------------
A non-text attachment was scrubbed...
Name: perf.zip
Type: application/zip
Size: 8346 bytes
Desc: not available
Url : http://www.lag.net/pipermail/paramiko/attachments/20061217/7ccdcb3f/attachment.zip