[paramiko] SFTP performance for large files

[Bill Zingler]

Hello Wouter,

I have attached my test scripts which I'm using for testing secure (scp,
sftp) and non-secure (ftp) file transfer clients on both Linux and Windows
to a Linux server.  All of my testing is being run over a dedicated Gigabit
Ethernet between the client and server.

I find that paramiko will run slower by a factor of 2 or 3 as compared to
scp on Linux and pscp on Windows for files that are less then 1 MB.  This is
quite reasonable for pure python implementation compared to a C or C++
implementation that scp and pscp are.  But as I tested file sizes in the 100
MB and 1 GB range, I find that paramiko will run slower by a factor of 20.
I'm sure that there is some room for optimization.

The testing is controlled by the .cfg file, so you can tailor it to change
file sizes and repetitions.  The script requires the ConfigObj module which
you can get from the Cheese Shop.  Other than that, the remaining
dependencies should be standard ones.

I would really like to use to use paramiko in my application and would if
only small files were needed to be transferred.

Thanks for the help,

Bill

On 12/13/06, Wouter van Heyst <larstiq at larstiq.dyndns.org> wrote:
>
> On Tue, Dec 12, 2006 at 03:25:36PM -0500, Bill Zingler wrote:
> > All,
> >
> > I have been doing some benchmarks on Linux for transferring files over
> SSH.
> > It appears that paramiko works relatively well for files less than 1 MB,
> but
> > for larger files the transfer time as compared to scp for 100 MB and 1
> GB
> > becomes 20 times longer.  Does anyone have any ideas as to why that is?
>
> Not yet, but in order to get a better grip on it, does that happen for
> individual files only, or also if you transfer 100 1MB sized files?
>
> Wouter van Heyst
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.6 (GNU/Linux)
>
> iQCVAwUBRX/7SenTGucjhaP+AQLQawP+KKcdfJLkaInQ7QrzCpnepKD0y/rO8Qjv
> frvIBDLhYtyakeOQbRLWyrGoqxWMz4lI742O7xB/sQ5pRB5s5L1Icbhk8Hzn272s
> mQT3+J7NzPvkJss+o+nDyyM7ZIFfeH/MVbiUNfFyDRjMlJ/IDj+Fz/h+FE5jpyZ9
> TreImQ+pRJA=
> =Vx+I
> -----END PGP SIGNATURE-----
>
>
> _______________________________________________
> paramiko mailing list
> paramiko at lag.net
> http://www.lag.net/mailman/listinfo/paramiko
>
>


-- 
Bill Zingler
Zingler & Associates, Inc.
(443) 822-4857
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.lag.net/pipermail/paramiko/attachments/20061214/25c3737a/attachment.htm 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: filetransferbenchmark.py
Type: text/x-python
Size: 10476 bytes
Desc: not available
Url : http://www.lag.net/pipermail/paramiko/attachments/20061214/25c3737a/attachment.py 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: filetransferbenchmark.cfg
Type: application/octet-stream
Size: 1195 bytes
Desc: not available
Url : http://www.lag.net/pipermail/paramiko/attachments/20061214/25c3737a/attachment.obj