From robey at lag.net Wed Feb 1 10:49:14 2006 From: robey at lag.net (Robey Pointer) Date: Wed Feb 1 11:58:38 2006 Subject: [paramiko] Paramiko and asyncore/asynchat In-Reply-To: <200602010009.22177.ralsina@kde.org> References: <200602010009.22177.ralsina@kde.org> Message-ID: <4FC13331-8FC9-4706-8CBE-26CAFE445C35@lag.net> On 31 Jan 2006, at 19:09, Roberto Alsina wrote: > Hello, I am playing with Paramiko, and trying to write a rather > unusual ssh > client. > > The main thing about it is that the ssh client is a separate > process from the > UI, and comunicates with it via asyncore.dispatcher objects. > > I had it working (kinda) using libssh, but I like paramiko better, > but I have > a problem. > > Is there a way to wrap the channel object into an asyncore > dispatcher? That > would make my code much simpler, but I am not understanding it :-( I believe you should be able to just create an asyncore dispatcher, passing a paramiko Channel instead of a socket. Channels are duck- typed to look as much like sockets as possible, including having a fileno() that lets select() work. > More details, in case it clears something: > > I have two processes, A and B. > > A is a GUI (PyQt) where you can choose what session you want to > create. > > B is a small terminal application. > > When you start a session on A, it opens a konsole running B, and A > connects to > the server using paramiko. > > Then B passes stdin to A using a socket, and A passes stdout and > stderr to B > using another socket. Like this: > > user->stdin->B->A->paramiko->server > server->stdout/stderr->paramiko->A->B->user > > A is polling the ssh server and B's sockets. > > Since I have the A<->B part working nicely using asyncore, I wanted > to wrap > A<->server, the paramiko connection, in the same way. > > I am currently doing that part using select.select as in the > example, and it > works, but it's kinda ugly ;-) It sounds pretty roundabout. Why bother to create the konsole? You could probably use subprocess or even just os.fork(). :) Anyway, I haven't tried using asyncore with paramiko, but I believe it should work. robey From robey at lag.net Wed Feb 1 11:34:16 2006 From: robey at lag.net (Robey Pointer) Date: Wed Feb 1 11:58:39 2006 Subject: [paramiko] Paramiko and EOF In-Reply-To: <152fd9bc0601261311p71e1816bic75925748190a85c@mail.gmail.com> References: <152fd9bc0601201333g44aad96dl34f7cc79fb3403af@mail.gmail.com> <4352A48D-F199-4039-B86C-BD29451608D2@lag.net> <152fd9bc0601230122u172ceff9m6556240758ea21a1@mail.gmail.com> <152fd9bc0601241101l171765ag7f2a69d92dfc5824@mail.gmail.com> <0878D44E-46B2-4719-9233-F3112A07F28C@lag.net> <152fd9bc0601261311p71e1816bic75925748190a85c@mail.gmail.com> Message-ID: On 26 Jan 2006, at 13:11, Thomas Steinacher wrote: >> You're sure this is with paramiko 1.5.2? There was a select() bug >> fixed in that release, and I still can't reproduce your bug using >> either python 2.3 or 2.4. > > Yes. Python 2.3 with Paramiko 1.5.2. I believe I found the EOF bug. It looks like Channel.recv is too eager to clear the select flag when the buffer is drained. Around line 515 of channel.py (in Channel.recv) you should see a line like this: if self.pipe is not None: # clear the pipe, since no more data is buffered self.pipe.clear() Try changing the condition to: if (self.pipe is not None) and not (self.closed or self.eof_received): # clear the pipe, since no more data is buffered self.pipe.clear() Once I was able to reproduce it, that change fixed it for me. Unfortunately I can't help with the 'event' problem. :( If there's a bug in 'event', it's beyond my ability to fix. If you can come up with a small enough reproducable case (you won't need paramiko to reproduce it), it's probably worth bringing up to the python- developers list. robey From robey at lag.net Wed Feb 1 17:02:41 2006 From: robey at lag.net (Robey Pointer) Date: Wed Feb 1 17:02:43 2006 Subject: [paramiko] exec_command returns false on Sun_SSH In-Reply-To: <43DACBB3.2090609@activestate.com> References: <43B46F40.6050004@activestate.com> <43DACBB3.2090609@activestate.com> Message-ID: <1E81AD56-A26B-4B12-993D-6273E36FD5AF@lag.net> On 27 Jan 2006, at 17:41, Todd Whiteman wrote: > Robey Pointer wrote: >> >> On 29 Dec 2005, at 15:20, Todd Whiteman wrote: >> >>> channel.exec_command() returns False even though the remote >>> command is successful and the data can be read through the channel. >>> >>> Is this an error with this verson of SSH (version info below)? >>> Do I just ignore this and use channel.recv_ready() as a better >>> indicator (which is set to true in this case)? >>> >>> $ ssh -V >>> Sun_SSH_1.1, SSH protocols 1.5/2.0, OpenSSL 0x0090704f >> >> Returning False but working would be a very odd thing to do: it >> looks like exec_command() will only return False if the channel is >> closed. And if the channel is closed, you wouldn't be able to >> read data from it. :) >> >> What version of paramiko are you using, and does the same thing >> happen in 1.5.2? >> >> robey > > This is with Paramiko 1.5.2, and I can read the data, which is kind > of strange. Maybe I have something incorrect in the my code (see > sample test code at end)... > > I can make the same problem happen on other platforms by adjusting > the sleep time and/or commenting out the channel.close(). With your sample code, I was able to reproduce the bug easily -- thanks! For short commands, sometimes the channel is closed before the exec_command thread wakes up to see the command is allowed. (This happens especially over the loopback socket.) The fix is to make Channel._wait_for_event (around line 1006 of channel.py) look like this: def _wait_for_event(self): while True: self.event.wait(0.1) if self.event.isSet(): break if self.closed: return False return True (This is also checked into the bzr tree.) robey From toddw at ActiveState.com Wed Feb 1 17:34:09 2006 From: toddw at ActiveState.com (Todd Whiteman) Date: Wed Feb 1 17:34:25 2006 Subject: [paramiko] exec_command returns false on Sun_SSH In-Reply-To: <1E81AD56-A26B-4B12-993D-6273E36FD5AF@lag.net> References: <43B46F40.6050004@activestate.com> <43DACBB3.2090609@activestate.com> <1E81AD56-A26B-4B12-993D-6273E36FD5AF@lag.net> Message-ID: <43E16191.2010007@activestate.com> Robey Pointer wrote: > > On 27 Jan 2006, at 17:41, Todd Whiteman wrote: > >> Robey Pointer wrote: >>> >>> On 29 Dec 2005, at 15:20, Todd Whiteman wrote: >>> >>>> channel.exec_command() returns False even though the remote command >>>> is successful and the data can be read through the channel. >>>> >>>> Is this an error with this verson of SSH (version info below)? >>>> Do I just ignore this and use channel.recv_ready() as a better >>>> indicator (which is set to true in this case)? >>>> >>>> $ ssh -V >>>> Sun_SSH_1.1, SSH protocols 1.5/2.0, OpenSSL 0x0090704f >>> >>> Returning False but working would be a very odd thing to do: it >>> looks like exec_command() will only return False if the channel is >>> closed. And if the channel is closed, you wouldn't be able to read >>> data from it. :) >>> >>> What version of paramiko are you using, and does the same thing >>> happen in 1.5.2? >>> >>> robey >> >> This is with Paramiko 1.5.2, and I can read the data, which is kind >> of strange. Maybe I have something incorrect in the my code (see >> sample test code at end)... >> >> I can make the same problem happen on other platforms by adjusting >> the sleep time and/or commenting out the channel.close(). > For short commands, sometimes the channel is closed before the > exec_command thread wakes up to see the command is allowed. (This > happens especially over the loopback socket.) > > The fix is to make Channel._wait_for_event (around line 1006 of > channel.py) look like this: > > def _wait_for_event(self): > while True: > self.event.wait(0.1) > if self.event.isSet(): > break > if self.closed: > return False > return True > > (This is also checked into the bzr tree.) > > robey > Cool! This must be only in the bzr repository, as I have no _wait_for_event() in paramiko 1.5.2 :) From ralsina at kde.org Thu Feb 2 03:00:00 2006 From: ralsina at kde.org (Roberto Alsina) Date: Thu Feb 2 02:55:56 2006 Subject: [paramiko] Paramiko and asyncore/asynchat In-Reply-To: <4FC13331-8FC9-4706-8CBE-26CAFE445C35@lag.net> References: <200602010009.22177.ralsina@kde.org> <4FC13331-8FC9-4706-8CBE-26CAFE445C35@lag.net> Message-ID: <200602020800.00984.ralsina@kde.org> El Mi?rcoles, 1 de Febrero de 2006 15:49, escribi?: > On 31 Jan 2006, at 19:09, Roberto Alsina wrote: > > Hello, I am playing with Paramiko, and trying to write a rather > > unusual ssh > > client. > > > > The main thing about it is that the ssh client is a separate > > process from the > > UI, and comunicates with it via asyncore.dispatcher objects. > > > > I had it working (kinda) using libssh, but I like paramiko better, > > but I have > > a problem. > > > > Is there a way to wrap the channel object into an asyncore > > dispatcher? That > > would make my code much simpler, but I am not understanding it :-( > > I believe you should be able to just create an asyncore dispatcher, > passing a paramiko Channel instead of a socket. Channels are duck- > typed to look as much like sockets as possible, including having a > fileno() that lets select() work. Almost, one small piece seems to be missing. *** Caught exception: exceptions.AttributeError: 'Channel' object has no attribute 'getpeername' Traceback (most recent call last): File "paramiko-async.py", line 85, in ? disp=asyncore.dispatcher(chan) File "/usr/lib/python2.3/asyncore.py", line 212, in __init__ self.addr = sock.getpeername() AttributeError: 'Channel' object has no attribute 'getpeername' If I add a dummy getpeername it works like a charm. I can probably even do a patch ;-) ? -- ?("\''/").__..-''"`-. . ? ? ? ? Roberto Alsina ?`9_ 9 ?) ? `-. ( ? ?).`-._.`) ?ralsina@kde.org ?(_Y_.)' ._ ? ) `._`. ?" -.-' ? KDE Developer (MFCH) ? _..`-'_..-_/ /-'_.' (l)-'' ((i).' ((!.' ? Buenos Aires - Argentina Imminentizing the eschaton since 1971. From ralsina at kde.org Thu Feb 2 03:35:53 2006 From: ralsina at kde.org (Roberto Alsina) Date: Thu Feb 2 03:31:43 2006 Subject: [paramiko] Example Paramiko and asyncore/asynchat In-Reply-To: <200602020800.00984.ralsina@kde.org> References: <200602010009.22177.ralsina@kde.org> <4FC13331-8FC9-4706-8CBE-26CAFE445C35@lag.net> <200602020800.00984.ralsina@kde.org> Message-ID: <200602020835.53928.ralsina@kde.org> Ok, here is a hackish version of how you can use paramiko and asyncore. The hackish comes from not handling close() correctly and a few other pieces, notably patching Channel to provide a getpeername on runtime ;-) but it does work almost like the simple demo. ? -- ?("\''/").__..-''"`-. . ? ? ? ? Roberto Alsina ?`9_ 9 ?) ? `-. ( ? ?).`-._.`) ?ralsina@kde.org ?(_Y_.)' ._ ? ) `._`. ?" -.-' ? KDE Developer (MFCH) ? _..`-'_..-_/ /-'_.' (l)-'' ((i).' ((!.' ? Buenos Aires - Argentina Imminentizing the eschaton since 1971. -------------- next part -------------- A non-text attachment was scrubbed... Name: paramiko-async.py Type: application/x-python Size: 4124 bytes Desc: not available Url : http://www.lag.net/pipermail/paramiko/attachments/20060202/e1b14309/paramiko-async.bin From philipp.knuesel at gmail.com Thu Feb 2 01:14:48 2006 From: philipp.knuesel at gmail.com (=?ISO-8859-1?Q?Philipp_Kn=FCsel?=) Date: Thu Feb 2 10:27:50 2006 Subject: [paramiko] paramiko 1.5.2 + pycrypto 2.0.1 win binary -> not compatible? In-Reply-To: References: Message-ID: Hello paramiko-fans I try to install paramiko 1.5.2 and pycrypto 2.0.1 (win binary for Python 2.4 from voidspace) on a Win XP Box After importing paramiko I get the following error message: File "C:\prog\Python24\Lib\site-packages\paramiko\util.py", line 31, in ? from paramiko.common import * File "C:\prog\Python24\Lib\site-packages\paramiko\common.py", line 106, in ? randpool = RandomPool() File "C:\prog\Python24\Lib\site-packages\Crypto\Util\randpool.py", line 82, in __init__ self._getPos = hash.digest_size AttributeError: 'module' object has no attribute 'digest_size' I'm new to python and don't know if I have to edit some code or is there an older pycrypto win binary which is compatible? Thank you very much! Philipp PS: please excuse incase this posting appears twice. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://www.lag.net/pipermail/paramiko/attachments/20060202/ca8ea44e/attachment.htm From thomas.st at gmail.com Fri Feb 3 06:01:33 2006 From: thomas.st at gmail.com (Thomas Steinacher) Date: Fri Feb 3 06:29:52 2006 Subject: [paramiko] Paramiko and EOF In-Reply-To: <152fd9bc0602030600q2436157elfd68430b1f9d19@mail.gmail.com> References: <152fd9bc0601201333g44aad96dl34f7cc79fb3403af@mail.gmail.com> <4352A48D-F199-4039-B86C-BD29451608D2@lag.net> <152fd9bc0601230122u172ceff9m6556240758ea21a1@mail.gmail.com> <152fd9bc0601241101l171765ag7f2a69d92dfc5824@mail.gmail.com> <0878D44E-46B2-4719-9233-F3112A07F28C@lag.net> <152fd9bc0601261311p71e1816bic75925748190a85c@mail.gmail.com> <152fd9bc0602030600q2436157elfd68430b1f9d19@mail.gmail.com> Message-ID: <152fd9bc0602030601r47ccad83i13070b9cee98022@mail.gmail.com> > Try changing the condition to: > > if (self.pipe is not None) and not (self.closed or > self.eof_received): > # clear the pipe, since no more data is buffered > self.pipe.clear() > > Once I was able to reproduce it, that change fixed it for me. It seems to work for me using your fix. I hope this will go into paramiko 1.5.3. > Unfortunately I can't help with the 'event' problem. :( If there's a > bug in 'event', it's beyond my ability to fix. If you can come up > with a small enough reproducable case (you won't need paramiko to > reproduce it), it's probably worth bringing up to the python- > developers list. Okay, I don't have a small case yet. Maybe there's a bug somewhere in PyCrypt. I think I will use multiple processes instead of threads in my application, because I don't have much time to debug this issue, and it should be faster on dual(-core) processors. Thank you for your help and for the EOF-fix. Thomas Steinacher From robey at lag.net Sun Feb 5 22:36:45 2006 From: robey at lag.net (Robey Pointer) Date: Sun Feb 5 22:36:52 2006 Subject: [paramiko] Example Paramiko and asyncore/asynchat In-Reply-To: <200602020835.53928.ralsina@kde.org> References: <200602010009.22177.ralsina@kde.org> <4FC13331-8FC9-4706-8CBE-26CAFE445C35@lag.net> <200602020800.00984.ralsina@kde.org> <200602020835.53928.ralsina@kde.org> Message-ID: On 2 Feb 2006, at 3:35, Roberto Alsina wrote: > Ok, here is a hackish version of how you can use paramiko and > asyncore. > > The hackish comes from not handling close() correctly and a few > other pieces, > notably patching Channel to provide a getpeername on runtime ;-) > but it does work almost like the simple demo. Thanks! I added a 'getpeernmae' method to paramiko's Channel object, so in the future it should work better with asyncore. robey From robey at lag.net Sun Feb 5 22:38:36 2006 From: robey at lag.net (Robey Pointer) Date: Sun Feb 5 22:38:40 2006 Subject: [paramiko] paramiko 1.5.2 + pycrypto 2.0.1 win binary -> not compatible? In-Reply-To: References: Message-ID: <6AB8CCE0-DD50-474B-BDCA-85887ABC1E02@lag.net> On 2 Feb 2006, at 1:14, Philipp Kn?sel wrote: > Hello paramiko-fans > > I try to install paramiko 1.5.2 and pycrypto 2.0.1 (win binary for > Python 2.4 from voidspace) > on a Win XP Box > > After importing paramiko I get the following error message: > > File "C:\prog\Python24\Lib\site-packages\paramiko\util.py", line > 31, in ? > from paramiko.common import * > File "C:\prog\Python24\Lib\site-packages\paramiko\common.py", > line 106, in ? > randpool = RandomPool() > File "C:\prog\Python24\Lib\site-packages\Crypto\Util > \randpool.py", line 82, in __init__ > self._getPos = hash.digest_size > AttributeError: 'module' object has no attribute 'digest_size' > > I'm new to python and don't know if I have to edit some code > or > is there an older pycrypto win binary which is compatible? That error appears to be coming from within pycrypto, so I suspect that pycrypto package is either broken or didn't install correctly. You might try a different distribution of pycrypto if you can find one. (Maybe a Windows user will have good advice here.) robey From erick_bodine at comcast.net Mon Feb 6 20:17:52 2006 From: erick_bodine at comcast.net (erick_bodine@comcast.net) Date: Mon Feb 6 20:33:07 2006 Subject: [paramiko] paramiko 1.5.2 + pycrypto 2.0.1 win binary -> not compatible? Message-ID: <020720060417.19724.43E81F700007751300004D0C22092299270A02070B010DA1050C079D0A@comcast.net> Have seen this before, something is wrong w/ the package or install order. Uninstall paramiko and pycrypto (double check Lib/site-packages), download pycrypto again, install pycrypto, install paramiko -- --ERick -------------- Original message ---------------------- From: Robey Pointer > > On 2 Feb 2006, at 1:14, Philipp Knüsel wrote: > > > Hello paramiko-fans > > > > I try to install paramiko 1.5.2 and pycrypto 2.0.1 (win binary for > > Python 2.4 from voidspace) > > on a Win XP Box > > > > After importing paramiko I get the following error message: > > > > File "C:\prog\Python24\Lib\site-packages\paramiko\util.py", line > > 31, in ? > > from paramiko.common import * > > File "C:\prog\Python24\Lib\site-packages\paramiko\common.py", > > line 106, in ? > > randpool = RandomPool() > > File "C:\prog\Python24\Lib\site-packages\Crypto\Util > > \randpool.py", line 82, in __init__ > > self._getPos = hash.digest_size > > AttributeError: 'module' object has no attribute 'digest_size' > > > > I'm new to python and don't know if I have to edit some code > > or > > is there an older pycrypto win binary which is compatible? > > That error appears to be coming from within pycrypto, so I suspect > that pycrypto package is either broken or didn't install correctly. > You might try a different distribution of pycrypto if you can find > one. (Maybe a Windows user will have good advice here.) > > robey > > > _______________________________________________ > paramiko mailing list > paramiko@lag.net > http://www.lag.net/mailman/listinfo/paramiko From philipp.knuesel at gmail.com Tue Feb 7 04:25:09 2006 From: philipp.knuesel at gmail.com (=?ISO-8859-1?Q?Philipp_Kn=FCsel?=) Date: Tue Feb 7 06:17:48 2006 Subject: [paramiko] paramiko 1.5.2 + pycrypto 2.0.1 win binary -> not compatible? In-Reply-To: <020720060417.19724.43E81F700007751300004D0C22092299270A02070B010DA1050C079D0A@comcast.net> References: <020720060417.19724.43E81F700007751300004D0C22092299270A02070B010DA1050C079D0A@comcast.net> Message-ID: > Uninstall paramiko and pycrypto (double check Lib/site-packages), download pycrypto again, install pycrypto, install paramiko > > -- > --ERick > Thank you Erick for your suggestion. Unfortunately it didn't work. I tried pycrypto 2.0.1 and 2.0 For the error message I assume there is an incompatibility. However, thank you for your point. Philipp From erick_bodine at comcast.net Tue Feb 7 08:11:37 2006 From: erick_bodine at comcast.net (erick_bodine@comcast.net) Date: Tue Feb 7 08:21:45 2006 Subject: [paramiko] paramiko 1.5.2 + pycrypto 2.0.1 win binary -> not compatible? Message-ID: <020720061611.25634.43E8C6B8000DFE6E0000642222007456720A02070B010DA1050C079D0A@comcast.net> You wouldn't by chance be trying to do this on an IBM ThinkPad, circa 2006-2004? The ThinkPads come with Python pre-installed to run some IBM laptop utilities. We have had problems at my place of work w/ getting 3rd party modules to run/install properly w/ this configuration. -- --ERick -------------- Original message ---------------------- From: Philipp Knüsel > > Uninstall paramiko and pycrypto (double check Lib/site-packages), download > pycrypto again, install pycrypto, install paramiko > > > > -- > > --ERick > > > > Thank you Erick for your suggestion. Unfortunately it didn't work. > I tried pycrypto 2.0.1 and 2.0 > For the error message I assume there is an incompatibility. > > However, thank you for your point. > > Philipp From robey at lag.net Tue Feb 7 10:39:27 2006 From: robey at lag.net (Robey Pointer) Date: Tue Feb 7 10:39:30 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <20060206080924.GE22467@home.puzzling.org> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> Message-ID: <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> I'm cc:ing the paramiko mailing list because I think this is relevant there too. On 6 Feb 2006, at 0:09, Andrew Bennetts wrote: > On Fri, Feb 03, 2006 at 10:41:11AM -0800, Robey Pointer wrote: >> >> On 2 Feb 2006, at 17:19, Andrew Bennetts wrote: >> >>> The root cause is that paramiko is using __del__. This is almost >>> always a bad >>> idea, because objects with __del__ cannot be garbage collected if >>> they're in a >>> reference cycle. It's an almost certain way to cause memory leaks, >>> unless >>> you're *extremely* careful to always break cycles manually. >>> >>> I think paramiko should not use __del__. >> >> Unfortunately __del__ is needed for some python-file emulation, which >> bzr and other apps rely on heavily, often without knowing (which is >> the point). But it should be okay as long as no cycles are created. >> But yes, I agree that use of __del__ is really a black art. > > What do you need to do on __del__? socket.sockets already close > themselves just > like files. I guess that was kind of vague, sorry. :) Python borrows a concept from c++, and causes files & sockets to auto-close themselves as soon as the last reference goes away. Since paramiko supplies both socket- like things (Channels) and file-like things (SFTPFiles), I've found that python coders expect them to have the same del behavior as builtins, allowing things like this: open('email.txt', 'w').write('jdoe@example.com') That said, I think the __del__ in Transport may be unnecessary. (I don't think a Transport can be collected until all its channels are already gone.) Though I'm still interested in cleaning up any remaining cycles that get created, because after reading threads on python-dev about this, I don't think the GC is going to be fixed anytime soon. > Cycles are really, really hard to avoid. If a traceback occurs, > that can create > cycles. There's lots of ways to unexpectedly make a cycle. How could a traceback create a cycle? > There's a pretty obvious cycle right here: > > self.packetizer.set_keepalive(interval, > lambda x=self: x.global_request('keepalive@lag.net', > wait=False)) Oops, yeah. Added to my to-do list. :) Although I don't think that's the one you're seeing, because this code isn't called by bzr. robey From mbp at sourcefrog.net Tue Feb 7 13:40:38 2006 From: mbp at sourcefrog.net (Martin Pool) Date: Tue Feb 7 14:03:15 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <1139347792.7153.33.camel@localhost.localdomain> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060207211741.GE6345@sourcefrog.net> <1139347792.7153.33.camel@localhost.localdomain> Message-ID: <20060207214038.GF6345@sourcefrog.net> On 8 Feb 2006, Robert Collins wrote: > On Wed, 2006-02-08 at 08:17 +1100, Martin Pool wrote: > > The traceback contains an object representing the stack, which contains > > a copy of all the local variables. This will cause additional > > references to objects, and might cause them to be retained for longer > > than you would expect. I'm not sure how it would cause a cycle passing > > through the file-like object, unless for some reason it saved the > > exception info within itself? > > By copy of the local variables, I presume you mean a copy of the local > names dictionary ;). No 'user' objects get copied. Or maybe another reference to the local names dictionary for each stack frame? At any rate objects referenced on the stack are "more reachable" by being referenced from the trackback object. -- Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://www.lag.net/pipermail/paramiko/attachments/20060208/b8a0f93b/attachment.pgp From mbp at sourcefrog.net Tue Feb 7 13:17:41 2006 From: mbp at sourcefrog.net (Martin Pool) Date: Tue Feb 7 14:03:23 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> Message-ID: <20060207211741.GE6345@sourcefrog.net> On 7 Feb 2006, Robey Pointer wrote: > I guess that was kind of vague, sorry. :) Python borrows a concept > from c++, and causes files & sockets to auto-close themselves as soon > as the last reference goes away. Since paramiko supplies both socket- > like things (Channels) and file-like things (SFTPFiles), I've found > that python coders expect them to have the same del behavior as > builtins, allowing things like this: > > open('email.txt', 'w').write('jdoe@example.com') Of course this statement is not really safe, and shouldn't be used in serious code. There's no guarantee that the file will be closed or its buffers flushed. I do write this myself all the time in throwaway scripts, and have never seen it fail in cPython 2.4. The only really safe way to do it is with a finally block that closes the file, and this is what we should do with sftp too. > >Cycles are really, really hard to avoid. If a traceback occurs, > >that can create > >cycles. There's lots of ways to unexpectedly make a cycle. > > How could a traceback create a cycle? The traceback contains an object representing the stack, which contains a copy of all the local variables. This will cause additional references to objects, and might cause them to be retained for longer than you would expect. I'm not sure how it would cause a cycle passing through the file-like object, unless for some reason it saved the exception info within itself? -- Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: Digital signature Url : http://www.lag.net/pipermail/paramiko/attachments/20060208/99252bb3/attachment.pgp From robertc at robertcollins.net Tue Feb 7 13:29:52 2006 From: robertc at robertcollins.net (Robert Collins) Date: Tue Feb 7 14:03:27 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <20060207211741.GE6345@sourcefrog.net> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060207211741.GE6345@sourcefrog.net> Message-ID: <1139347792.7153.33.camel@localhost.localdomain> On Wed, 2006-02-08 at 08:17 +1100, Martin Pool wrote: > The traceback contains an object representing the stack, which contains > a copy of all the local variables. This will cause additional > references to objects, and might cause them to be retained for longer > than you would expect. I'm not sure how it would cause a cycle passing > through the file-like object, unless for some reason it saved the > exception info within itself? By copy of the local variables, I presume you mean a copy of the local names dictionary ;). No 'user' objects get copied. Rob -- GPG key available at: . -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part Url : http://www.lag.net/pipermail/paramiko/attachments/20060208/c48e77b5/attachment.pgp From toddw at ActiveState.com Tue Feb 7 15:44:15 2006 From: toddw at ActiveState.com (Todd Whiteman) Date: Tue Feb 7 15:44:27 2006 Subject: [paramiko] exec_command returns false on Sun_SSH In-Reply-To: <1E81AD56-A26B-4B12-993D-6273E36FD5AF@lag.net> References: <43B46F40.6050004@activestate.com> <43DACBB3.2090609@activestate.com> <1E81AD56-A26B-4B12-993D-6273E36FD5AF@lag.net> Message-ID: <43E930CF.1000903@activestate.com> Robey Pointer wrote: > For short commands, sometimes the channel is closed before the > exec_command thread wakes up to see the command is allowed. (This > happens especially over the loopback socket.) > > The fix is to make Channel._wait_for_event (around line 1006 of > channel.py) look like this: > > def _wait_for_event(self): > while True: > self.event.wait(0.1) > if self.event.isSet(): > break > if self.closed: > return False > return True I have now had a chance to test this and it is now working correctly. Good work :) Thanks, Todd From andrew at canonical.com Tue Feb 7 17:17:13 2006 From: andrew at canonical.com (Andrew Bennetts) Date: Wed Feb 8 15:08:45 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> Message-ID: <20060208011713.GG547@home.puzzling.org> On Tue, Feb 07, 2006 at 10:39:27AM -0800, Robey Pointer wrote: [...] > > > >What do you need to do on __del__? socket.sockets already close > >themselves just > >like files. > > I guess that was kind of vague, sorry. :) Python borrows a concept > from c++, and causes files & sockets to auto-close themselves as soon > as the last reference goes away. Since paramiko supplies both socket- > like things (Channels) and file-like things (SFTPFiles), I've found > that python coders expect them to have the same del behavior as > builtins, allowing things like this: > > open('email.txt', 'w').write('jdoe@example.com') But why do you need to do anything explicitly in paramiko to support this? The only external resource you have (that I'm aware of) is a socket, and python's socket type already takes care of closing that when the last reference goes away. By implementing __del__, you can actually make the situation worse: the transport might never be collected (because it might be in a cycle), so its socket is never reclaimed, thus causing a file descriptor leak (as well as the obvious memory leak). So, I guess I'm missing what finalisation behaviour you want to happen that doesn't already happen without __del__? Sending some sort of polite "quit" message to the server rather than just closing the socket? In some cases it can be possible to replace __del__ with weakref callbacks (which don't have this problem), although I can't say whether that's a possibility here without understanding what you're using __del__ for :) Another clever trick can be to replace the __del__ on your main object with a __del__ on an attribute of that object that is never referenced elsewhere. Twisted's Deferred objects do this -- for various reasons, it really helps to log when a Deferred with an unhandled error is collected. We used to have a __del__ on Deferred, which then would tend to cause some Deferreds to get trapped in gc.garbage (and their warnings about unhandled errors never to be logged, either, because they were never collected) -- and it was always hard to predict if any given Deferred would be in a cycle or not. Now they have a "_debugInfo" attribute, which holds a "DebugInfo" object with a __del__. The DebugInfo object has no reference to its Deferred. The Deferred will update the DebugInfo as appropriate, so that it can always log the right warning (if any). The DebugInfo object has a __del__, but is extremely simple and only ever referenced by one object, and only has references to simple strings. Deferreds no longer have a __del__, so there's no problem with them being in cycles. > That said, I think the __del__ in Transport may be unnecessary. (I > don't think a Transport can be collected until all its channels are > already gone.) Though I'm still interested in cleaning up any > remaining cycles that get created, because after reading threads on > python-dev about this, I don't think the GC is going to be fixed > anytime soon. Yeah, I don't expect the GC in CPython to change significantly any time soon. > >Cycles are really, really hard to avoid. If a traceback occurs, > >that can create > >cycles. There's lots of ways to unexpectedly make a cycle. > > How could a traceback create a cycle? See the warning at the top of http://www.python.org/doc/lib/inspect-stack.html A traceback raised from inside paramiko and caught by a caller might be used in all sorts of ways that paramiko can't know about -- including keeping a reference to it (for later inspection, perhaps). So even though paramiko itself might not be creating cycles that it won't break, callers of paramiko can easily make other cycles with paramiko inadvertently. > >There's a pretty obvious cycle right here: > > > > self.packetizer.set_keepalive(interval, > > lambda x=self: x.global_request('keepalive@lag.net', > >wait=False)) > > Oops, yeah. Added to my to-do list. :) Although I don't think > that's the one you're seeing, because this code isn't called by bzr. Yeah, I don't think this is my culprit either. But ensuring that your cycle avoidance/breaking is perfect seems like much harder work than not using __del__... -Andrew. From andrew at canonical.com Wed Feb 8 08:14:56 2006 From: andrew at canonical.com (Andrew Bennetts) Date: Wed Feb 8 15:08:46 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <20060206080924.GE22467@home.puzzling.org> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060208011713.GG547@home.puzzling.org> <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> Message-ID: <20060208161456.GP547@home.puzzling.org> On Mon, Feb 06, 2006 at 07:09:24PM +1100, Andrew Bennetts wrote: > On Fri, Feb 03, 2006 at 10:41:11AM -0800, Robey Pointer wrote: [...] > > > > Do you have any way to reproduce this outside of Zope? I tried > > running the bzr selftests (and a few other tests, like 'bzr push' to > > an sftp site) with 'python -i' and then doing gc.collect() and > > looking in gc.garbage, but wasn't able to reproduce it. > > I'm working on this. I'll let you know when I've got something minimal and > complete I can share with you. I have some progress to report. The garbage occurs during my tests' tearDown. Specifically, after I stop the SFTP server process. This SFTP server is Twisted Conch-based. At the time tear down happens, there's a live Branch object from the test hanging around (the result of a bzrlib.branch.Branch.open('sftp://localhost...')), which is of course still connected. One definite result of this set up is that when the server goes away, the Packetizer.read_all function raises EOFError in the Transport's thread. You might be able to reproduce this yourself by simulating this. I've also found that this code snippet will clear the garbage for me: for gbg in gc.garbage: if hasattr(gbg, 'auth_handler'): gbg.auth_handler = None del gbg del gc.garbage[:] gc.collect() -Andrew. From robey at lag.net Wed Feb 8 21:33:01 2006 From: robey at lag.net (Robey Pointer) Date: Wed Feb 8 21:33:09 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <20060208161456.GP547@home.puzzling.org> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060208011713.GG547@home.puzzling.org> <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <20060208161456.GP547@home.puzzling.org> Message-ID: On 8 Feb 2006, at 8:14, Andrew Bennetts wrote: > On Mon, Feb 06, 2006 at 07:09:24PM +1100, Andrew Bennetts wrote: >> On Fri, Feb 03, 2006 at 10:41:11AM -0800, Robey Pointer wrote: > [...] >>> >>> Do you have any way to reproduce this outside of Zope? I tried >>> running the bzr selftests (and a few other tests, like 'bzr push' to >>> an sftp site) with 'python -i' and then doing gc.collect() and >>> looking in gc.garbage, but wasn't able to reproduce it. >> >> I'm working on this. I'll let you know when I've got something >> minimal and >> complete I can share with you. > > I have some progress to report. Thanks! That was enough to help me reproduce it. And fix it. It turned out to be a paper-bag bug where the auth_handler was pretty blatantly creating a cycle. I'm embarrased. :) (Oddly, Packetizer wasn't involved, so its presence in gc.garbage was a red herring.) I fixed the cycle by using a weakref, then I removed Transport's __del__ since it doesn't need one anyway. The bzr branch for paramiko is , so if you want a quick fix, you should be able to pull it from there. robey From andrew at canonical.com Wed Feb 8 21:44:39 2006 From: andrew at canonical.com (Andrew Bennetts) Date: Wed Feb 8 21:44:45 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060208011713.GG547@home.puzzling.org> <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <20060208161456.GP547@home.puzzling.org> Message-ID: <20060209054439.GU547@home.puzzling.org> On Wed, Feb 08, 2006 at 09:33:01PM -0800, Robey Pointer wrote: > > On 8 Feb 2006, at 8:14, Andrew Bennetts wrote: > > >On Mon, Feb 06, 2006 at 07:09:24PM +1100, Andrew Bennetts wrote: > >>On Fri, Feb 03, 2006 at 10:41:11AM -0800, Robey Pointer wrote: > >[...] > >>> > >>>Do you have any way to reproduce this outside of Zope? I tried > >>>running the bzr selftests (and a few other tests, like 'bzr push' to > >>>an sftp site) with 'python -i' and then doing gc.collect() and > >>>looking in gc.garbage, but wasn't able to reproduce it. > >> > >>I'm working on this. I'll let you know when I've got something > >>minimal and > >>complete I can share with you. > > > >I have some progress to report. > > Thanks! That was enough to help me reproduce it. And fix it. It > turned out to be a paper-bag bug where the auth_handler was pretty > blatantly creating a cycle. I'm embarrased. :) (Oddly, Packetizer > wasn't involved, so its presence in gc.garbage was a red herring.) Well, the Packetizer instances are referenced from the uncollectable Transports, hence they were also uncollectable. Being uncollectable and having __del__ means Python will put it in gc.garbage, as described at http://docs.python.org/lib/module-gc.html#l2h-410 > I fixed the cycle by using a weakref, then I removed Transport's > __del__ since it doesn't need one anyway. The bzr branch for > paramiko is , so if you > want a quick fix, you should be able to pull it from there. Thanks! While we're on the subject, why does Packetizer have a __del__? Sockets take care of closing themselves when collected. -Andrew. From robey at lag.net Wed Feb 8 22:17:55 2006 From: robey at lag.net (Robey Pointer) Date: Wed Feb 8 22:18:01 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <20060208011713.GG547@home.puzzling.org> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060208011713.GG547@home.puzzling.org> Message-ID: <52C472C3-DFB1-47D7-839F-101B35421D16@lag.net> On 7 Feb 2006, at 17:17, Andrew Bennetts wrote: > On Tue, Feb 07, 2006 at 10:39:27AM -0800, Robey Pointer wrote: > [...] >>> >>> What do you need to do on __del__? socket.sockets already close >>> themselves just >>> like files. >> >> I guess that was kind of vague, sorry. :) Python borrows a concept >> from c++, and causes files & sockets to auto-close themselves as soon >> as the last reference goes away. Since paramiko supplies both >> socket- >> like things (Channels) and file-like things (SFTPFiles), I've found >> that python coders expect them to have the same del behavior as >> builtins, allowing things like this: >> >> open('email.txt', 'w').write('jdoe@example.com') > > But why do you need to do anything explicitly in paramiko to > support this? The > only external resource you have (that I'm aware of) is a socket, > and python's > socket type already takes care of closing that when the last > reference goes > away. Right. The problem comes when SFTPFile objects (for example) are used the same way. The sftp session may hang around for a while, but the coder who uses the idiom above expects the file to be flushed and closed as soon as the "file object" goes out of scope. BTW, I agree with those earlier in the thread who said this idiom is playing a little fast & loose. But it's used a lot even in the bzr source, so don't throw too many stones. ;) > So, I guess I'm missing what finalisation behaviour you want to > happen that > doesn't already happen without __del__? Sending some sort of > polite "quit" > message to the server rather than just closing the socket? In some cases, the file may not have finished writing its data yet until del'd or closed. This is to sidestep latency issues when pushing a lot of data. I know some of this may seem arcane & obfuscated. I think it's important to take complex things and present them through a simple interface, so I go through some hoops to make ssh channels behave as much like sockets as possible (even to the point where you can select () on them), etc, even though it often makes the underlying implementation harder. > In some cases it can be possible to replace __del__ with weakref > callbacks > (which don't have this problem), although I can't say whether that's a > possibility here without understanding what you're using __del__ > for :) [omitted clever tricks which I saved because they're pretty cool] It doesn't look like I can use either the weakref callback (which I didn't know about till today) or the trick of adding a __del__ to a member variable, because in the destructor I'll need to effectively call "close()" on the destroyed object, which won't be available from either place. >>> Cycles are really, really hard to avoid. If a traceback occurs, >>> that can create >>> cycles. There's lots of ways to unexpectedly make a cycle. >> >> How could a traceback create a cycle? > > See the warning at the top of http://www.python.org/doc/lib/inspect- > stack.html > > A traceback raised from inside paramiko and caught by a caller > might be used in > all sorts of ways that paramiko can't know about -- including > keeping a > reference to it (for later inspection, perhaps). So even though > paramiko itself > might not be creating cycles that it won't break, callers of > paramiko can > easily make other cycles with paramiko inadvertently. I see how that could make paramiko objects have a longer lifetime than expected, but not how it could create cycles. Assuming I kept references in paramiko from having cycles (ie, made them a DAG), outside sources should be able to make references to anything inside, without breaking the DAG. Sorry if this was too long/geeky. I tried to go back and edit it down. robey From andrew at canonical.com Wed Feb 8 23:31:35 2006 From: andrew at canonical.com (Andrew Bennetts) Date: Wed Feb 8 23:31:41 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <52C472C3-DFB1-47D7-839F-101B35421D16@lag.net> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060208011713.GG547@home.puzzling.org> <52C472C3-DFB1-47D7-839F-101B35421D16@lag.net> Message-ID: <20060209073135.GW547@home.puzzling.org> [Dropped bazaar-ng from CC list, we're talking purely about paramiko now, I think] On Wed, Feb 08, 2006 at 10:17:55PM -0800, Robey Pointer wrote: > > On 7 Feb 2006, at 17:17, Andrew Bennetts wrote: > [...] > >But why do you need to do anything explicitly in paramiko to support this? > >The only external resource you have (that I'm aware of) is a socket, and > >python's socket type already takes care of closing that when the last > >reference goes away. > > Right. The problem comes when SFTPFile objects (for example) are > used the same way. The sftp session may hang around for a while, but > the coder who uses the idiom above expects the file to be flushed and > closed as soon as the "file object" goes out of scope. I still don't think __del__ is necessary for this. :) Looking at SFTPFile objects, I see the following actions are performed by your __del__ (which simply invokes self.close()): - wait for outstanding responses from the server, if any. - flush buffered writes, if any. - send CMD_CLOSE to the server. [please correct me if I've missed something!] For the outstanding responses to an unreferenced file, you could just discard them. I see your code already deals with this case, if a file has been closed before a response was received. For buffered writes, you don't want them to be lost. That's fair enough. You could keep a "must_complete" dictionary in your SFTPClient to keep a reference to SFTPFiles that need to exist, even if all other code has stopped referencing them. When the write completes, the SFTPFile can be unregistered from the dict. To send the CMD_CLOSE, you don't need the SFTPFile instance. A weakref with a callback of sftpclient._request(CMD_CLOSE, handle) kept in the SFTPFile can do this. E.g. keep track of the open handles in mapping in your SFTPClient along with the weakref callback, when a close request is sent, remove the handle from self.handles to avoid sending CMD_CLOSEs for the same handle twice. Here's a sketch: in class SFTPFile: def __init__(self, sftp, handle, mode='r', bufsize=-1): ... sftp.register_handle(handle, self) def close(self): ... self.sftp.close_handle(self.handle) in SFTPClient: def __init__(self, sock): ... self._handles_to_close = {} def register_handle(self, handle, file) fileref = weakref.ref(file, lambda: self.close_handle(handle)) self._handles_to_close[handle] = fileref def close_handle(self, handle): del self._handles_to_close[handle] self._request(CMD_CLOSE, handle) > BTW, I agree with those earlier in the thread who said this idiom is > playing a little fast & loose. But it's used a lot even in the bzr > source, so don't throw too many stones. ;) Right :) I appreciate you want to make paramiko work as smoothly as possible, even for lazy programmers. I'm sometimes lazy, so I have no argument with this :) I think it's possible to satisfy this without using __del__, though... > >So, I guess I'm missing what finalisation behaviour you want to happen that > >doesn't already happen without __del__? Sending some sort of polite "quit" > >message to the server rather than just closing the socket? > > In some cases, the file may not have finished writing its data yet > until del'd or closed. This is to sidestep latency issues when > pushing a lot of data. Right, someone might do sftpclient.open('foo', 'w').write(a_gigabyte). > I know some of this may seem arcane & obfuscated. I think it's > important to take complex things and present them through a simple > interface, so I go through some hoops to make ssh channels behave as > much like sockets as possible (even to the point where you can select > () on them), etc, even though it often makes the underlying > implementation harder. So you shouldn't mind contortions to avoid __del__ then, to behave more like sockets that never end up in gc.garbage ;) > >In some cases it can be possible to replace __del__ with weakref callbacks > >(which don't have this problem), although I can't say whether that's a > >possibility here without understanding what you're using __del__ for :) > > [omitted clever tricks which I saved because they're pretty cool] > > It doesn't look like I can use either the weakref callback (which I > didn't know about till today) or the trick of adding a __del__ to a > member variable, because in the destructor I'll need to effectively > call "close()" on the destroyed object, which won't be available from > either place. The trick I think is to decompose close() into smaller operations, as I suggest above. [...] > >A traceback raised from inside paramiko and caught by a caller might be used > >in all sorts of ways that paramiko can't know about -- including keeping a > >reference to it (for later inspection, perhaps). So even though paramiko > >itself might not be creating cycles that it won't break, callers of paramiko > >can easily make other cycles with paramiko inadvertently. > > I see how that could make paramiko objects have a longer lifetime > than expected, but not how it could create cycles. Assuming I kept > references in paramiko from having cycles (ie, made them a DAG), > outside sources should be able to make references to anything inside, > without breaking the DAG. Outside sources could choose to make cycles, though -- and expect it to work without any problems, because most python objects work fine in cycles. The fact that your objects aren't collectable in cycles is in some ways part of the API. Most of the time, Python programmers can be blissfully ignorant of the garbage collector, but libraries that use __del__ methods can force them to care. I'll admit that taking paramiko objects and storing them in cycles is a bit obscure, but in a sufficiently complex program I wouldn't be surprised if it could be an issue. -Andrew. From john at arbash-meinel.com Thu Feb 9 04:30:13 2006 From: john at arbash-meinel.com (John A Meinel) Date: Thu Feb 9 10:39:29 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <52C472C3-DFB1-47D7-839F-101B35421D16@lag.net> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060208011713.GG547@home.puzzling.org> <52C472C3-DFB1-47D7-839F-101B35421D16@lag.net> Message-ID: <43EB35D5.5060805@arbash-meinel.com> Robey Pointer wrote: > > On 7 Feb 2006, at 17:17, Andrew Bennetts wrote: > ... >> In some cases it can be possible to replace __del__ with weakref >> callbacks >> (which don't have this problem), although I can't say whether that's a >> possibility here without understanding what you're using __del__ for :) > > [omitted clever tricks which I saved because they're pretty cool] > > It doesn't look like I can use either the weakref callback (which I > didn't know about till today) or the trick of adding a __del__ to a > member variable, because in the destructor I'll need to effectively call > "close()" on the destroyed object, which won't be available from either > place. > You could move the close() into the member variable. And then main_obj.close() just calls the main_obj._secret_sauce.close(). It just depends what needs to happen. John =:-> -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 249 bytes Desc: OpenPGP digital signature Url : http://www.lag.net/pipermail/paramiko/attachments/20060209/9021be44/signature-0001.pgp From robey at lag.net Sun Feb 12 19:20:35 2006 From: robey at lag.net (Robey Pointer) Date: Sun Feb 12 19:20:41 2006 Subject: [paramiko] Re: Possible bug in sftp_client.py In-Reply-To: <1139423473.43ea38f14a402@webmail.welho.com> References: <1139423473.43ea38f14a402@webmail.welho.com> Message-ID: <3BB1C3DC-6ED3-4D9D-82FC-83A4F6A27A66@lag.net> On 8 Feb 2006, at 10:31, vpohjol3@welho.com wrote: > I was playing around with paramiko-1.5.2 and noticed that you get > an exception > if you do an lsdir on directory that contains filesnames that > contain non-ascii > characters, such as ?, ? or ? (theese are in latin-1). I was afraid this day would come... The SFTP standard mandates that all filenames are in utf8. But on many platforms (like unix), the sftp server has almost zero chance of knowing the actual encoding of filenames on the system -- they're just bytes. I was afraid one day I'd run into a server that doesn't bother with the attempt, and just sends the unencoded bytes, utf8 or not. Guess that finally happened. :) > I had to change line 37 of sftp_client.py from > > return s.decode('utf-8') > > to > > return s.decode('latin-1') > > to get it working for me (this is probably not a good solution). I wonder what a good solution would be. I suppose if the filename isn't ascii, but can't be decoded into utf8, we should just punt and return the filename as a byte string. We certainly have no hope of guessing the correct encoding if the server can't. Something like this: def _to_unicode(s): try: return s.encode('ascii') except UnicodeError: try: return s.decode('utf-8') except UnicodeError: return s Would that work okay for your purposes? It passes the unit tests for me, at least. robey From robey at lag.net Mon Feb 13 10:20:12 2006 From: robey at lag.net (Robey Pointer) Date: Mon Feb 13 10:58:09 2006 Subject: [paramiko] Re: gc.garbage from paramiko In-Reply-To: <20060209073135.GW547@home.puzzling.org> References: <20060203011959.GA31363@home.puzzling.org> <4DED4722-A172-4D18-981E-3EC7C5E52599@lag.net> <20060206080924.GE22467@home.puzzling.org> <2C1113D4-ECAE-4F6A-BA08-B9C8045A4D65@lag.net> <20060208011713.GG547@home.puzzling.org> <52C472C3-DFB1-47D7-839F-101B35421D16@lag.net> <20060209073135.GW547@home.puzzling.org> Message-ID: <26C7C18C-8966-4ECE-94C6-67603D9D7229@lag.net> On 8 Feb 2006, at 23:31, Andrew Bennetts wrote: > I still don't think __del__ is necessary for this. :) > > Looking at SFTPFile objects, I see the following actions are > performed by > your __del__ (which simply invokes self.close()): > > - wait for outstanding responses from the server, if any. > - flush buffered writes, if any. > - send CMD_CLOSE to the server. > > [please correct me if I've missed something!] I think that's correct. I worked on this a bit last night, but ran into a problem that stopped me cold. (Maybe you can help?) The BufferedFile.__del__ function flushes the write buffer, which may be filled by opening the file with a write buffer >= 0. (This doesn't appear to be common, but it is part of the python file API.) It flushes the buffer by calling _write() on itself, which is intended to be overridden by subclasses. The net effect is that __del__ really needs a reference to itself or it may lose data. I think your comments about SFTPFile.__del__ would work, so it could be made to work fine without a __del__. But since SFTPFile is a subclass of BufferedFile, I don't think removing its __del__ will matter if BufferedFile still has one, so now I'm back to being inclined to leave it alone. robey From robey at lag.net Sun Feb 19 23:48:32 2006 From: robey at lag.net (Robey Pointer) Date: Sun Feb 19 23:48:38 2006 Subject: [paramiko] release: paramiko 1.5.3 Message-ID: <783C7458-A8A7-4C8C-9353-31500EB6E615@lag.net> Since it's been awhile, here's another paramiko release: http://www.lag.net/paramiko/ Mostly this just contains bugfixes, a couple of performance improvements, and a few new features in SFTPClient/SFTPFile. The full update list from the README: v1.5.3 SQUIRTLE * a few performance enhancements * added HostKeys, for dealing with openssh style "known_hosts" files, and added support for hashed hostnames * added Transport.atfork() for dealing with forked children * added SFTPClient.truncate, SFTPFile.chmod, SFTPFile.chown, SFTPFile.utime, and SFTPFile.truncate * improved windows demos [patch from mike looijmans], added an sftp demo, and moved demos to the demos/ folder * fixed a few interoperability bugs * cleaned up logging a bit * fixed a bug where EOF on a Channel might not be detected by select [found by thomas steinacher] * fixed python 2.4-ism that crept in [patch by jan hudec] * fixed a few reference loops that could have interacted badly with the python garbage collector * fixed a bunch of pychecker warnings, some of which were bugs Squirtle really needs no introduction. He's the best pok?mon in the universe, and blows away stupid ole Pikachu. robey