From burak.arslan at arskom.com.tr Mon May 5 22:55:46 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Tue, 06 May 2014 01:55:46 +0300 Subject: [spyne] test Message-ID: <536816F2.1010501@arskom.com.tr> hello world From burak.arslan at arskom.com.tr Thu May 8 14:07:39 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Thu, 08 May 2014 17:07:39 +0300 Subject: [spyne] Testing 2.10 code with 2.11 Message-ID: <536B8FAB.8020009@arskom.com.tr> Hello all and welcome to the Spyne mailing list :) As I said earlier, if everything goes according to plan, I'm planning to make a Spyne release in around a month. While I'd prefer to make sure that updating Spyne won't break already working code, with almost 1300 commits spanning a year of active development, my impression is that this has very low probability of happening, especially because Spyne 2.11 is way stricter in terms of object definition consistency. Even though the changelog is supposed to summarize https://github.com/arskom/spyne/compare/2_10...master even that is getting too long to be wieldy. So here's a list of possible things that can break existing code: 1) Schema non-determinism due to inheritance: Spyne now adds all child classes of a given parent class to the xml schema document, regardless of whether it's used in service definitions. This is a first step towards implementing polymorphic responses. When a subclass contains a field that is also present in the parent class, you will see a "content model not determinist" error and the server will refuse to boot. Cleaning duplicate fields will fix this issue. 2) Unequal number of parameters in @rpc and function definition: Spyne 2.10 did not care when @rpc had more arguments than the actual function definition. Spyne 2.11 won't tolerate this. 3) Spyne 2.11 uses a proper topological sorting algorithm for generating the xml schema. So the order of object declarations will certainly be different from what 2.10 generates. This is not supposed to cause any issues though. 4) The field order within the object definitions should *in theory* stay the same, but we never know as CPython offers no guarantees about the element order consistency in its hashmap implementation. Some folks found this so unacceptable that they dared to submit this horrendous hack: https://github.com/arskom/spyne/pull/343. As it seemed to work OK and after making sure that it's strictly opt-in, I chose to merge it. It's been in the tree for a couple of weeks now and seems to be working OK with CPython 2.6-3.3 and PyPy. Please test it with your environment and report back as it's relying, as far as I can tell, on some implementation details of CPython. Even though these are technically backwards-incompatible changes, they're also bug fixes that detect broken code. I don't want to allow broken code solely on backwards-compatibility grounds, so I intend make sure these changes go into Spyne 2.11. Anything else that breaks existing code, please let me know. This post was about changes to the stable parts of Spyne. As you might have guessed, a lot of effort also went to adding new functionality to Spyne but I'll talk about these in a separate post that focuses on new features. Thanks for using Spyne. I hope you find the new version useful. Best regards, Burak From burak.arslan at arskom.com.tr Thu May 8 19:00:20 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Thu, 08 May 2014 22:00:20 +0300 Subject: [spyne] robots@spyne.io Message-ID: <536BD444.8060303@arskom.com.tr> Hi, FYI, There's now another mailing list that receives notifications from all the robots tending to Spyne's source code. Currently the list receives: 1) Commit notifications from github 2) Test notifications from travis 3) Test notifications from Jenkins installation at http://spyne.ci.cloudbees.com The web interface is at: http://lists.spyne.io/listinfo/robots Best regards, Burak From rduldulao at chikka.com Tue May 13 04:50:42 2014 From: rduldulao at chikka.com (Rodolfo Narciso Duldulao Jr.) Date: Tue, 13 May 2014 12:50:42 +0800 Subject: [spyne] Help with Spyne HttpRpc Message-ID: <4088341940-17008@mail.chikka.com> Hello all, Spyne newbie here. ?I am trying to find a way to limit HttpRpc requests to use POST request method only. ?We have a web service that according to a spec, should be accessible via HTTP POST only. ?I have made a decorator to check for the REQUEST_METHOD value like so: class OurWebService(ServiceBase):? ? @rpc(Unicode, _returns=Iterable(String))? ? @require_methods(['POST'])? ? def say_hello(ctx, name):? ? ? ? ?yield "Hello %s." % name ? ? ?#from the example Here's the code for the require_methods decorator: class require_methods(object):? ? def __init__(self, methods):? ? ? ? assert isinstance(methods, list), "Methods should be in list form."? ? ? ? self.methods = methods ? ? def __call__(self, f): ? ? ? ? def wrapped_f(*args, **kwargs):? ? ? ? ? ? assert args[0], "No context param"? ? ? ? ? ? assert getattr(args[0], 'in_document'), "No context info"? ? ? ? ? ? if args[0].in_document.get('REQUEST_METHOD') not in self.methods:? ? ? ? ? ? ? ? raise InvalidMethodError()? ? ? ? ? ? pprint(args)? ? ? ? ? ? f(*args, **kwargs) ? ? ? ? wrapped_f.__doc__ = f.__doc__? ? ? ? return wrapped When I access the web service via GET, I get an Error 500 and I see the InvalidMethodError stack trace logged on my console - which is to be expected.When I use POST (using a REST Console), it says these on the debug output: DEBUG:spyne.protocol.http: body ? : {'name': ['Dave']}DEBUG:spyne.protocol.dictdoc:discarding field 'name'... (Stack trace here with lines coming from the decorator code) TypeError: say_hello() takes exactly 2 arguments (1 given) It seems to me, the rpc decorator didn't recognize the name parameter now that the service method is wrapped by another decorator. ?Am I doing the correct approach? ?Any advice is appreciated.? Thanks, Rodolfo N. Duldulao, Jr.Chikka Philippines, Inc. Penthouse,23 ADB Avenue, Ortigas Center,Pasig City 1605 Metro Manila,?Philippines? Email: rduldulao at chikka.com?Mobile: 09209515026 Tel: 029889629 -------------- next part -------------- An HTML attachment was scrubbed... URL: From burak.arslan at arskom.com.tr Tue May 13 08:09:48 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Tue, 13 May 2014 11:09:48 +0300 Subject: [spyne] Help with Spyne HttpRpc In-Reply-To: <4088341940-17008@mail.chikka.com> References: <4088341940-17008@mail.chikka.com> Message-ID: <5371D34C.2000301@arskom.com.tr> Hi there, On 05/13/14 07:50, Rodolfo Narciso Duldulao Jr. wrote: > Hello all, > > Spyne newbie here. I am trying to find a way to limit HttpRpc > requests to use POST request method only. We have a web service that > according to a spec, should be accessible via HTTP POST For (the not yet released) 2.11, pass _patterns=[HttpPattern(verb='POST')] to @rpc. For 2.10, this could work: _patterns=[HttpPattern('/say_hello', verb='POST')] though HttpPattern had a lot of issues in 2.10, so if it blows up in an unexpected way, just give up. > > It seems to me, the rpc decorator didn't recognize the name parameter > now that the service method is wrapped by another decorator. Am I > doing the correct approach? Any advice is appreciated. > http://spyne.io/docs/2.9/faq.html#is-it-possible-to-use-other-decorators-with-rpc-srpc Best, Burak From burak.arslan at arskom.com.tr Mon May 19 13:40:31 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Mon, 19 May 2014 16:40:31 +0300 Subject: [spyne] [Soap-Python] soap library for python-3.x In-Reply-To: References: <20140425095615.M32856@cdot.in> <536CE6A1.6000209@oss.schwarz.eu> Message-ID: <537A09CF.6060803@arskom.com.tr> Hello All, And as you may know, I maintain Spyne. I also wrote most of it. Also, correct me if I'm wrong here, Spyne is the oldest maintained Soap codebase written in Python. It's also WS-I compliant. First, this long post won't conclude that Spyne is best and we should all use it. I'll say something different. Please read on. I'm sure we all agree that implementing a SOAP library is a huge undertaking. Implementing SOAP really means implementing Xml Schema, Wsdl 1.1 Wsdl 2.0 Soap 1.1 and Soap 1.2 standards, both in a client and server setting. Implementing daemons in Python is also a minefield. Depending on the project requirements, you need to use gevent, twisted or WSGI and learn about quirks of its many implementations etc. Without a proper abstraction in place, it is very difficult to switch between these. Spyne is not just a Soap library anymore, it also became that abstraction. Because that's what I had fun working on. There's more than 1000 commit difference between 2.10 and 2.11. There's a lot more goodies in there but lower level efforts make up the bigger part of the 2.11 effort. So if you go your your own way and re-implement Soap from the ground up, you'll run into the issues we already ran into, and you'll have to deal with them just like we had to. Perhaps you'll end up with a more complete Soap implementation than Spyne, but you'll also have a less robust RPC codebase than Spyne. A couple of concrete examples: 1) For an xsd:sequence (the default class declaration method in both soapbox and spyne) to pass validation, the tags in it must be *in the same order it's defined in the wsdl document*. I'm looking at this example and seeing the same naive mistake that we did in Spyne: https://github.com/FelixSchwarz/soapbox-bsd/blob/master/examples/xml_complex_types.py The field order in the sequence is defined by the dict passed to xsd.ComplexType metaclass. The order in python dicts are *random* and change according to a wild array of factors, especially now that we have PYTHONHASHSEED=random as an option. So when you add an innocent new field to an existing object declaration, be ready to have all cached wsdl documents invalidated. See https://github.com/arskom/spyne/pull/313 and https://github.com/arskom/spyne/pull/343 to see how we solved that problem. Fixing it in Python 2 wasn't easy! 2) what are your precautions for parsing XML documents from untrusted sources? https://github.com/FelixSchwarz/soapbox-bsd/blob/31aff9f5825f1d6520a84bd938663851a45b5a03/soapbox/xsd.py#L859 None, from what I can tell. This part of spyne documentation is full of security-related xml links: http://spyne.io/docs/2.10/reference/protocol/xml.html#spyne.protocol.xml._base.XmlDocument The most famous one: https://en.wikipedia.org/wiki/Billion_laughs 3) Are you using incremental xml generation? Can you produce a 8 gigabyte xml document without having it all in memory with SoapBox? Apparently not: https://github.com/FelixSchwarz/soapbox-bsd/blob/31aff9f5825f1d6520a84bd938663851a45b5a03/soapbox/xsd.py#L920 With Spyne 2.11, you can, thanks to lxml's incremental generation api. I could go on and on. (The above points also apply to PySimpleSoap but I'm don't want to compare it with Spyne as PySimpleSoap's code is really naive and seemingly deliberately so as it says simple right there in the project's name. I feel the two projects serve two very different audiences. ) So my point is: If you go and make rude statements like this: On 05/09/14 17:30, Felix Schwarz wrote: > (Disclaimer: I'm one of the maintainers of "soapbox-bsd".) > > I think you should check out soapbox-bsd which is the only real option for > server-side SOAP in Python: https://github.com/FelixSchwarz/soapbox-bsd you're not doing open source. (only *real* option? come on, Felix Schwarz, you shouldn't be so full of yourself) Open source is about collaboration. What I'm seeing so far is not a willingness to collaborate but wasteful chest-pounding fed by ignorance about the delicacies of our craft I'm sadly used to see in corporate environments. You may or may not like Spyne, or PySimpleSoap or, soapbox, (or Ladon, it did Soap with Python last time I looked as well). None of them are perfect. But you must recognise that 1) everybody will benefit from unifying these efforts. 2) our lives will be made easier once the iniatial integration works are done with, because now we're a team. So here's what I suggest: Let's put our egos aside, look at each other's code, identify the strengths of each project, make sacrifices and unify Soap efforts in Python. If doing my part will mean doing a git rm -r spyne/protocol/soap or git rm -r spyne/interface/wsdl and simply import an adapter for SoapBox when importing spyne.protocol.soap, i'll do it, provided it will result in a better Soap experience for Spyne users. So, awaiting your feedback. Best regards, Burak From jcasale at activenetwerx.com Fri May 23 22:00:47 2014 From: jcasale at activenetwerx.com (Joseph L. Casale) Date: Fri, 23 May 2014 22:00:47 +0000 Subject: [spyne] accessing the wsdl Message-ID: I am dealing with what appears to be a broken client that is accessing a service I wrote. Does spyne have a facility to control where the wsdl is exposed? As a result of current setup, I am unable to leverage redirect/rewrites for this. Thanks, jlc From burak.arslan at arskom.com.tr Sat May 24 10:09:41 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Sat, 24 May 2014 13:09:41 +0300 Subject: [spyne] accessing the wsdl In-Reply-To: References: Message-ID: <53806FE5.7030809@arskom.com.tr> Hello Joseph, On 24/05/14 01:00, Joseph L. Casale wrote: > I am dealing with what appears to be a broken client that is accessing a service I wrote. > > Does spyne have a facility to control where the wsdl is exposed? As a result of current > setup, I am unable to leverage redirect/rewrites for this. You can use the wsdl event of the WsgiApplication wsgi_app = WsgiApplication(...) def _on_wsdl(ctx): print ctx.transport.wsdl # set this to whatever you want wsgi_app.event_manager.add_listener('wsdl', _on_wsdl) Best regards, Burak From burak.arslan at arskom.com.tr Mon May 26 16:06:13 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Mon, 26 May 2014 18:06:13 +0200 Subject: [spyne] accessing the wsdl In-Reply-To: <4b9f91f5aee04a7d9b7faa1f11c07ed3@exch.activenetwerx.com> References: , <53806FE5.7030809@arskom.com.tr> <4b9f91f5aee04a7d9b7faa1f11c07ed3@exch.activenetwerx.com> Message-ID: <53836675.50001@arskom.com.tr> On 26/05/14 16:54, Joseph L. Casale wrote: >>> I am dealing with what appears to be a broken client that is accessing a service I wrote. >>> >>> Does spyne have a facility to control where the wsdl is exposed? As a result of current >>> setup, I am unable to leverage redirect/rewrites for this. >> You can use the wsdl event of the WsgiApplication >> >> wsgi_app =sgiApplication(...) >> >> def _on_wsdl(ctx): >> print ctx.transport.wsdl # set this to whatever you want >> >> wsgi_app.event_manager.add_listener('wsdl', _on_wsdl) > Hey Burak, > I looked into this and the wsgi class but I think I failed to convey to my issue very well. > > While I am testing with a client for which I have no control nor any insight from its > esteemed operator, I notice an older implementation I am replacing returns the wsdl > on a 404 instead of an xml error for example as this silly client makes all sorts of > unexpected requests in addition to the one I am interested in responding to. > > I looked through the docs and code for the wsgi class and it wasn't immediately obvious > to me as to how to accomplish this. And this time you lost me. Last time I looked at that spec was a long time ago, but AFAIR SOAP-over-HTTP spec says servers should return either 200, 401 or 500, you're not allowed to return anything else. So returning wsdl document on 404 is quite weird and frankly I can't think of an easy way of adding that functionality easily. If you want to alter when WsgiApplication returns the wsdl document, just subclass it and override is_wsdl_request. You might end up copying the whole (wsdl) logic and altering it to match the behaviour you need though. Fortunately, wsdl handling code in WsgiApplication is not that complicated. Btw, This might be the weirdest SOAP quirk I've ever heard :) I hope I was able to help. Best regards, Burak From burak.arslan at arskom.com.tr Fri May 30 06:40:31 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Fri, 30 May 2014 09:40:31 +0300 Subject: [spyne] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <53879C80.2000009@pke.hr> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> <538270C2.5010901@arskom.com.tr> <53879C80.2000009@pke.hr> Message-ID: <538827DF.7020901@arskom.com.tr> On 29/05/14 23:45, Jurko Gospodneti? wrote: > Hi Arslan. > Hello, My name's Burak. Arslan is my surname. So call me burak :) > On 26.5.2014. 0:37, Burak Arslan wrote: >> See: >> >> https://spyne.ci.cloudbees.com/job/spyne/PYFLAV=2.7/lastCompletedBuild/testReport/spyne.test.interop.test_suds/TestSuds/ >> >> https://github.com/arskom/spyne/blob/master/spyne/test/interop/test_suds.py >> >> >> Spyne is migrating towards tox (Spyne was there way before tox was >> around) but the suds interop tests are not there yet. So I can't have >> two versions of suds tests that runs against both suds and suds-jurko >> yet. >> >> But you can run Spyne tests against your suds locally. Just clone Spyne >> repo and edit setup.py to use suds-jurko instead of suds here: >> https://github.com/arskom/spyne/blob/master/setup.py#L344 > > I've just tried to clone your project and run its tests using > suds-jurko, but I seem to be running into trouble with it so here's > some information on what I have & did. Perhaps you'll be able to > provide me with some more insight. > > * cloned from https://github.com/arskom/spyne.git into > 'D:\Workplace\Python Spyne' > * have Python 2.7.6 x64 installation with setuptools 3.6 > * CPython 2.7.6 installed in 'C:\Program Files\Python\Python276' > * just in case paths with spaces might be causing problems, I > linked the Spyne project location to 'Y:\' and the Python > installation to 'X:\'. > * I run 'X:\python.exe setup.py test' from the 'Y:\' folder and > loads of packages get installed after which the tests run. > > The first batch of tests runs but there the following tests fail: > >> spyne\test\test_util.py:197: in test_log_repr_complex >>> assert log_repr(val) == "Z(z=['abc', 'abc', 'abc', 'abc', >>> (...)])" >> E AssertionError: assert 'Z(z=[])' == "Z(z=['abc', 'abc', >> 'abc', 'abc', (...)])" >> E - Z(z=[]) >> E + Z(z=['abc', 'abc', 'abc', 'abc', (...)]) >> >> spyne\test\test_util.py:177: in test_log_repr_simple >>> assert log_repr(['a','b','c'], Array(String)) == "['a', 'b', >>> 'c']" >> E AssertionError: assert '[]' == "['a', 'b', 'c']" >> E - [] >> E + ['a', 'b', 'c'] > > They also result in the following two exceptions: > Yeah, Travis thought the tests passed because of the issue you filed yesterday. I fixed that and I'll fix the tests asap. As for the following issues you reported, it is mostly because you're running things on windows. Actually, you might be the first person ever running Spyne test suite on windows. >> Exception in thread Thread-6: >> Traceback (most recent call last): >> File "X:\lib\threading.py", line 810, in __bootstrap_inner >> self.run() >> File "X:\lib\threading.py", line 763, in run >> self.__target(*self.__args, **self.__kwargs) >> File "X:\lib\multiprocessing\pool.py", line 329, in _handle_workers >> debug('worker handler exiting') >> TypeError: 'NoneType' object is not callable >> >> Exception in thread Thread-7: >> Traceback (most recent call last): >> File "X:\lib\threading.py", line 810, in __bootstrap_inner >> self.run() >> File "X:\lib\threading.py", line 763, in run >> self.__target(*self.__args, **self.__kwargs) >> File "X:\lib\multiprocessing\pool.py", line 353, in _handle_tasks >> debug('task handler got sentinel') >> TypeError: 'NoneType' object is not callable > > After that tox failed miserably, which turned out to be caused by > Python installation being located in the root folder. Actually I think > it's virtualenv's fault but I did not do any significant debugging > here. Once I relocated it to 'X:\Python276' instead of 'X:\' and reran > the tests using 'X:\Python276\python.exe setup.py test' - tox started > to install django into its virtual environments and all the django > tests were reported as passed (and d*mn those django installations > take a looooong time :-D). > yes, but just the first time :) > After that I don't think your 'call_pytest_subprocess' tests work > quite the way you'd want them, at least not on Windows - whichever one > I try to run I get the following error: > >> Traceback (most recent call last): >> File "setup.py", line 438, in >> 'test_multi_python': RunMultiPythonTests >> File "X:\Python276\lib\distutils\core.py", line 152, in setup >> dist.run_commands() >> File "X:\Python276\lib\distutils\dist.py", line 953, in run_commands >> self.run_command(cmd) >> File "X:\Python276\lib\distutils\dist.py", line 972, in run_command >> cmd_obj.run() >> File "build\bdist.win-amd64\egg\setuptools\command\test.py", line >> 146, in run >> File "build\bdist.win-amd64\egg\setuptools\command\test.py", line >> 127, in with_project_on_sys_path >> File "setup.py", line 248, in run_tests >> capture=self.capture) or ret >> File "setup.py", line 126, in call_pytest_subprocess >> return call_test(pytest.main, args, tests) >> File "setup.py", line 64, in call_test >> p.start() >> File "X:\Python276\lib\multiprocessing\process.py", line 130, in start >> self._popen = Popen(self) >> File "X:\Python276\lib\multiprocessing\forking.py", line 277, in >> __init__ >> dump(process_obj, to_child, HIGHEST_PROTOCOL) >> File "X:\Python276\lib\multiprocessing\forking.py", line 199, in dump >> ForkingPickler(file, protocol).dump(obj) >> File "X:\Python276\lib\pickle.py", line 224, in dump >> self.save(obj) >> File "X:\Python276\lib\pickle.py", line 331, in save >> self.save_reduce(obj=obj, *rv) >> File "X:\Python276\lib\pickle.py", line 419, in save_reduce >> save(state) >> File "X:\Python276\lib\pickle.py", line 286, in save >> f(self, obj) # Call unbound method with explicit self >> File "X:\Python276\lib\pickle.py", line 649, in save_dict >> self._batch_setitems(obj.iteritems()) >> File "X:\Python276\lib\pickle.py", line 681, in _batch_setitems >> save(v) >> File "X:\Python276\lib\pickle.py", line 286, in save >> f(self, obj) # Call unbound method with explicit self >> File "X:\Python276\lib\pickle.py", line 748, in save_global >> (obj, module, name)) >> pickle.PicklingError: Can't pickle > 0x0000000003FDCAC8>: it's not found as __main__._ > You might know that Windows doesn't have the fork() syscall. Multiprocessing relies heavily on fork() to work on unix. You know what it does on windows? Starts a new Python process, pickles EVERYTHING, sends them to the new process and tries to reconstruct the state in the "child" process to simulate a fork. So to use multiprocessing in a cross platform way, you need to make sure EVERYTHING is pickleable. In my book, that's just insane, so I didn't bother with windows. If anybody's got a better idea for calling py.test from the setup script, I'm open to suggestions. https://docs.python.org/2/library/multiprocessing.html#windows > And the main test suite seems to exit and leave behind a background > runner process that exits like this: > >> Traceback (most recent call last): >> File "", line 1, in >> File "X:\Python276\lib\multiprocessing\forking.py", line 380, in main >> prepare(preparation_data) >> File "X:\Python276\lib\multiprocessing\forking.py", line 495, in >> prepare >> '__parents_main__', file, path_name, etc >> File "Y:\setup.py", line 438, in >> 'test_multi_python': RunMultiPythonTests >> File "X:\Python276\lib\distutils\core.py", line 152, in setup >> dist.run_commands() >> File "X:\Python276\lib\distutils\dist.py", line 953, in run_commands >> self.run_command(cmd) >> File "X:\Python276\lib\distutils\dist.py", line 972, in run_command >> cmd_obj.run() >> File "build\bdist.win-amd64\egg\setuptools\command\test.py", line >> 146, in run >> File "build\bdist.win-amd64\egg\setuptools\command\test.py", line >> 127, in with_project_on_sys_path >> File "Y:\setup.py", line 248, in run_tests >> capture=self.capture) or ret >> File "Y:\setup.py", line 126, in call_pytest_subprocess >> return call_test(pytest.main, args, tests) >> File "Y:\setup.py", line 64, in call_test >> p.start() >> File "X:\Python276\lib\multiprocessing\process.py", line 130, in start >> self._popen = Popen(self) >> File "X:\Python276\lib\multiprocessing\forking.py", line 258, in >> __init__ >> cmd = get_command_line() + [rhandle] >> File "X:\Python276\lib\multiprocessing\forking.py", line 358, in >> get_command_line >> is not going to be frozen to produce a Windows executable.''') >> RuntimeError: >> Attempt to start a new process before the current process >> has finished its bootstrapping phase. >> >> This probably means that you are on Windows and you have >> forgotten to use the proper idiom in the main module: >> >> if __name__ == '__main__': >> freeze_support() >> ... >> >> The "freeze_support()" line can be omitted if the program >> is not going to be frozen to produce a Windows executable. > > If you have any ideas, I'd love to get this working. :-D > Well, we need a call_pytest_subprocess that works on windows. Probably one that doesn't use multiprocessing. You can just do py.test test_suds.py though, it should work. > Oh, one more test left - the final one, called using the > 'call_trial' function: > > 1. It requires the pywin32 package to be installed in the target > Python environment (Twisted uses the win32api module module), and > setup does not install that itself. And it's a bitchy module to > install automatically since it comes as a windows installer and can > not be automatically installed using pip. :-( > > 2. When you do install pywin32 it spurts out lots of debug & info log > output which I have no idea what to do with. > > 3. Among that ton of text the following error message gets repeated a > few times which I also have no idea what to do with or whether it even > represents a test failure or not: > >> ERROR:spyne.application:Possible >> Traceback (most recent call last): >> File "y:\spyne\application.py", line 144, in process_request >> ctx.out_object = self.call_wrapper(ctx) >> File "y:\spyne\application.py", line 199, in call_wrapper >> retval = ctx.descriptor.service_class.call_wrapper(ctx) >> File "y:\spyne\service.py", line 207, in call_wrapper >> return ctx.function(*ctx.in_object) >> File "y:\spyne\test\interop\server\_service.py", line 336, in >> python_exception >> raise Exception("Possible") >> Exception: Possible > This is output from a test to see how twisted handles unexpected exceptions. So this is not an error. > 4. And similar to 3. the following error appears once: > >> ERROR:spyne.application:Fault(Plausible: 'A plausible fault') >> Traceback (most recent call last): >> File "y:\spyne\application.py", line 144, in process_request >> ctx.out_object = self.call_wrapper(ctx) >> File "y:\spyne\application.py", line 199, in call_wrapper >> retval = ctx.descriptor.service_class.call_wrapper(ctx) >> File "y:\spyne\service.py", line 207, in call_wrapper >> return ctx.function(*ctx.in_object) >> File "y:\spyne\test\interop\server\_service.py", line 340, in >> soap_exception >> raise Fault("Plausible", "A plausible fault", >> 'http://faultactor.example.com') >> Fault: Fault(Plausible: 'A plausible fault') > And this is testing how twisted handles "expected" exceptions. So this also is not an error. Thank you for your detailed explanations. I'm adding a "running on windows" section to the test suite readme and update it according to your input. Best regards, Burak From burak.arslan at arskom.com.tr Fri May 30 06:43:07 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Fri, 30 May 2014 09:43:07 +0300 Subject: [spyne] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <53821039.5030806@pke.hr> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> Message-ID: <5388287B.6090601@arskom.com.tr> On 25/05/14 18:46, Jurko Gospodneti? wrote: > I'd be most grateful if you could use the fork in any sort of a test > suite you have set up for your project - the more test coverage it > gets, the better. Especially considering that the original project had > not a single automated tests. See: https://spyne.ci.cloudbees.com/job/spyne/PYFLAV=2.7/lastCompletedBuild/testReport/spyne.test.interop.test_suds/TestSuds/ https://github.com/arskom/spyne/blob/master/spyne/test/interop/test_suds.py Spyne is migrating towards tox (Spyne was there way before tox was around) but the suds interop tests are not there yet. So I can't have two versions of suds tests that runs against both suds and suds-jurko yet. But you can run Spyne tests against your suds locally. Just clone Spyne repo and edit setup.py to use suds-jurko instead of suds here: https://github.com/arskom/spyne/blob/master/setup.py#L344 If you're planning to change the root namespace (e.g. suds => jsuds) I can simply copy test_suds.py to test_jsuds.py so both could be tested side by side. I also contemplated taking over suds some time ago but Suds codebase was not that adorable last time I looked at it. See: http://stackoverflow.com/a/15156236 The fork I mention there is: https://github.com/plq/suds Lastly, suds is *very* slow, because its XML parsing is pure-python. Also, it can't do async because unlike Spyne, transports are not pluggable. If, instead of improving suds, you're willing to look at Spyne's SOAP client, I can help you much more there. It does do Xml Schema parsing but not Wsdl parsing, otherwise it's more or less working. Good luck with your efforts, you seem to be doing a great job and great service to the community. Best regards, Burak From dsuch at zato.io Fri May 30 06:55:57 2014 From: dsuch at zato.io (Dariusz Suchojad) Date: Fri, 30 May 2014 08:55:57 +0200 Subject: [spyne] [Soap-Python] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <5388287B.6090601@arskom.com.tr> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> <5388287B.6090601@arskom.com.tr> Message-ID: <53882B7D.7050507@zato.io> On 05/30/2014 08:43 AM, Burak Arslan wrote: [About suds] > Also, it can't do async because unlike Spyne, transports are not > pluggable. Just for the record - I use suds-jurko asynchronously in Zato, which is based on gunicorn/gevent, just fine. I can't say whether the original suds library could have been used because I didn't really investigate that option, went straight to suds-jurko. The only issue I had was that suds-jurko inherited from plain suds class/client-level caches that had to be worked around by obtaining a queue of pre-configured number of SOAP clients in their own queues by an initializing Zato server. https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/http_soap/outgoing.py#L314 https://github.com/zatosource/zato/blob/master/code/zato-server/src/zato/server/connection/queue.py So essentially: - gevent monkey patches everything suds-jurko needs thus making the latter asynchronous - Zato creates a queue of client connections - Users can run suds-jurko in an asynchronous manner - Everyone is happy :-) cheers, -- Dariusz Suchojad https://zato.io ESB, SOA, REST, APIs and cloud integrations in Python From jurko.gospodnetic at pke.hr Fri May 30 08:02:25 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Fri, 30 May 2014 10:02:25 +0200 Subject: [spyne] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <538827DF.7020901@arskom.com.tr> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> <538270C2.5010901@arskom.com.tr> <53879C80.2000009@pke.hr> <538827DF.7020901@arskom.com.tr> Message-ID: <53883B11.7080401@pke.hr> Hi Burak. On 30.5.2014. 8:40, Burak Arslan wrote: > On 29/05/14 23:45, Jurko Gospodneti? wrote: > My name's Burak. Arslan is my surname. So call me burak :) Oh, woops... :-) Btw. tried to look you up on Skype before, but there a ton of 'Burak Arslans' there. :-D >> They also result in the following two exceptions: > > Yeah, Travis thought the tests passed because of the issue you filed > yesterday. I fixed that and I'll fix the tests asap. Hehe, glad to hear that helped. :-D > As for the following issues you reported, it is mostly because you're > running things on windows. Actually, you might be the first person ever > running Spyne test suite on windows. Woops again... :-D Unfortunately my day-job is mostly Windows related so I do not have a Linux based machine easily available. :-( But with or without that, I'd really like for suds and its testing/development tools to be OS agnostic. :-) >> After that I don't think your 'call_pytest_subprocess' tests work >> quite the way you'd want them, at least not on Windows - whichever one >> I try to run I get the following error: >> >>> Traceback (most recent call last): >>>[...snipped...] >>> pickle.PicklingError: Can't pickle >> 0x0000000003FDCAC8>: it's not found as __main__._ >> > > You might know that Windows doesn't have the fork() syscall. > Multiprocessing relies heavily on fork() to work on unix. You know what > it does on windows? Starts a new Python process, pickles EVERYTHING, > sends them to the new process and tries to reconstruct the state in the > "child" process to simulate a fork. > > So to use multiprocessing in a cross platform way, you need to make sure > EVERYTHING is pickleable. In my book, that's just insane, so I didn't > bother with windows. If anybody's got a better idea for calling py.test > from the setup script, I'm open to suggestions. > > https://docs.python.org/2/library/multiprocessing.html#windows Yup, I know about the forking issues on Windows and I'll try to take a closer look and get it working as I get enough free time, this weekend if I'm lucky :-). Is it just needed to run external pytest processes? If that is so, I guess the same could be achieved using the subprocess module. Or perhaps I can just track down the non-picklable parts and see if they can be tweaked a bit. The 'unpicklable function' mentioned in the traceback smells like it's a simple matter of local vs. module-level function usage or something like that. >> And the main test suite seems to exit and leave behind a background >> runner process that exits like this: >> >>> Traceback (most recent call last): >>>[...snipped...] >> >> If you have any ideas, I'd love to get this working. :-D > > Well, we need a call_pytest_subprocess that works on windows. Probably > one that doesn't use multiprocessing. > > You can just do py.test test_suds.py though, it should work. Ok, I'll run the tests using pytest directly and see how that goes. >> Oh, one more test left - the final one, called using the >> 'call_trial' function: >> >> 2. When you do install pywin32 it spurts out lots of debug & info log >> output which I have no idea what to do with. >> >> 3. Among that ton of text the following error message gets repeated a >> few times which I also have no idea what to do with or whether it even >> represents a test failure or not: >> >>> ERROR:spyne.application:Possible >>>Traceback (most recent call last): >>>[...snipped...] > > This is output from a test to see how twisted handles unexpected > exceptions. So this is not an error. > >> 4. And similar to 3. the following error appears once: >> >>> ERROR:spyne.application:Fault(Plausible: 'A plausible fault') >>> Traceback (most recent call last): >>>[...snipped...] > > And this is testing how twisted handles "expected" exceptions. So this > also is not an error. Testing all this makes sense, but I'd suggest implementing that test run to not display its output so verbosely. Specific test output should be displayed to the user only if the test fails and possibly not even then for such a general all-encompassing test suite such as the one run using 'setup.py test'. Best regards, Jurko Gospodneti? From burak.arslan at arskom.com.tr Fri May 30 08:17:18 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Fri, 30 May 2014 11:17:18 +0300 Subject: [spyne] [Soap-Python] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <53882B7D.7050507@zato.io> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> <5388287B.6090601@arskom.com.tr> <53882B7D.7050507@zato.io> Message-ID: <53883E8E.5080705@arskom.com.tr> On 30/05/14 09:55, Dariusz Suchojad wrote: > On 05/30/2014 08:43 AM, Burak Arslan wrote: > > [About suds] > >> Also, it can't do async because unlike Spyne, transports are not >> pluggable. > Just for the record - I use suds-jurko asynchronously in Zato, which is > based on gunicorn/gevent, just fine. Well, you're correct. Gevent does give you a sync interface on top of an async foundation. But when I said async, I was thinking of twisted-like event driven programming. I should have said "suds can't be used in an event-driven way". After having heard what Guido said about greenlet (look up his pycon 2013 keynote), I must have totally erased it from my memory :) And I agree with him you know, I won't rely on the insanity of monkeypatching the whole stdlib in order to do task switching in userspace: https://github.com/python-greenlet/greenlet/tree/master/platform Which also doesn't invalidate the pluggable transports point. Correct me if I'm wrong but last time I looked, you couldn't do Soap over ZeroMQ, msgpack or twisted with suds. I need to admit though: Spyne's client bits are not as complete as I'd like. But I hope to eventually offer client support in Spyne that is on par with server support. And it just would be so nice if somebody helped me with that! Best regards, Burak From burak.arslan at arskom.com.tr Fri May 30 08:38:16 2014 From: burak.arslan at arskom.com.tr (Burak Arslan) Date: Fri, 30 May 2014 11:38:16 +0300 Subject: [spyne] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <53883B11.7080401@pke.hr> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> <538270C2.5010901@arskom.com.tr> <53879C80.2000009@pke.hr> <538827DF.7020901@arskom.com.tr> <53883B11.7080401@pke.hr> Message-ID: <53884378.4070101@arskom.com.tr> On 30/05/14 11:02, Jurko Gospodneti? wrote: > Is it just needed to run external pytest processes? Yep. > If that is so, I guess the same could be achieved using the subprocess > module. That's what I tried first but found out that finding out which version of py.test executable to run using subprocess was problematic when you have multiple CPython versions installed. So i used multiprocessing to call pytest's main function. Maybe this whole state transfer thing could be avoided, or there could be a better way of locating python package entry points. For your case though, just running py.test test_suds.py should be enough. Best regards, Burak From dsuch at zato.io Fri May 30 08:37:59 2014 From: dsuch at zato.io (Dariusz Suchojad) Date: Fri, 30 May 2014 10:37:59 +0200 Subject: [spyne] [Soap-Python] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <53883E8E.5080705@arskom.com.tr> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> <5388287B.6090601@arskom.com.tr> <53882B7D.7050507@zato.io> <53883E8E.5080705@arskom.com.tr> Message-ID: <53884367.3000408@zato.io> On 05/30/2014 10:17 AM, Burak Arslan wrote: > Well, you're correct. Gevent does give you a sync interface on top of an > async foundation. But when I said async, I was thinking of twisted-like > event driven programming. I should have said "suds can't be used in an > event-driven way". > > After having heard what Guido said about greenlet (look up his pycon > 2013 keynote), I must have totally erased it from my memory :) And I > agree with him you know, I won't rely on the insanity of monkeypatching > the whole stdlib in order to do task switching in userspace: > https://github.com/python-greenlet/greenlet/tree/master/platform > > Which also doesn't invalidate the pluggable transports point. Correct me > if I'm wrong but last time I looked, you couldn't do Soap over ZeroMQ, > msgpack or twisted with suds. Sure, naturally - I've just mentioned it in case one day someone attempts to find information on whether suds-jurko is greenlet/gevent-safe, which is a nice way to make an application asynchronous. With a caveat on its thread/greenlet-safety - and this can be worked around by a dedicate pool of connections - this can be done. cheers, -- Dariusz Suchojad https://zato.io ESB, SOA, REST, APIs and cloud integrations in Python From jurko.gospodnetic at pke.hr Fri May 30 10:28:08 2014 From: jurko.gospodnetic at pke.hr (=?UTF-8?B?SnVya28gR29zcG9kbmV0acSH?=) Date: Fri, 30 May 2014 12:28:08 +0200 Subject: [spyne] How keep Python 3 moving forward - suds & Python 3 In-Reply-To: <53884378.4070101@arskom.com.tr> References: <5381AFD8.4050106@pke.hr> <5381EB66.8030105@arskom.com.tr> <53821039.5030806@pke.hr> <538270C2.5010901@arskom.com.tr> <53879C80.2000009@pke.hr> <538827DF.7020901@arskom.com.tr> <53883B11.7080401@pke.hr> <53884378.4070101@arskom.com.tr> Message-ID: <53885D38.60305@pke.hr> Hi Burak. On 30.5.2014. 10:38, Burak Arslan wrote: >> If that is so, I guess the same could be achieved using the subprocess >> module. > > That's what I tried first but found out that finding out which version > of py.test executable to run using subprocess was problematic when you > have multiple CPython versions installed. So i used multiprocessing to > call pytest's main function. > > Maybe this whole state transfer thing could be avoided, or there could > be a better way of locating python package entry points. Actually you can just run sys.executable with "-m pytest" command-line options. :-) Then there is no need to go hunting for the "py.test" executable. As I said before - I'll take a stab at cleaning that up and get back to you. Best regards, Jurko Gospodneti? P.S. Could you register the people at spyne.io list with gmane? I really prefer the NNTP interface.