[spyne] How keep Python 3 moving forward - suds & Python 3

Burak Arslan burak.arslan at arskom.com.tr
Fri May 30 06:40:31 UTC 2014


On 29/05/14 23:45, Jurko Gospodnetić wrote:
>   Hi Arslan.
>

Hello,

My name's Burak. Arslan is my surname. So call me burak :)


> On 26.5.2014. 0:37, Burak Arslan wrote:
>> See:
>>
>> https://spyne.ci.cloudbees.com/job/spyne/PYFLAV=2.7/lastCompletedBuild/testReport/spyne.test.interop.test_suds/TestSuds/ 
>>
>> https://github.com/arskom/spyne/blob/master/spyne/test/interop/test_suds.py 
>>
>>
>> Spyne is migrating towards tox (Spyne was there way before tox was
>> around) but the suds interop tests are not there yet. So I can't have
>> two versions of suds tests that runs against both suds and suds-jurko 
>> yet.
>>
>> But you can run Spyne tests against your suds locally. Just clone Spyne
>> repo and edit setup.py to use suds-jurko instead of suds here:
>> https://github.com/arskom/spyne/blob/master/setup.py#L344
>
>   I've just tried to clone your project and run its tests using 
> suds-jurko, but I seem to be running into trouble with it so here's 
> some information on what I have & did. Perhaps you'll be able to 
> provide me with some more insight.
>
> * cloned from https://github.com/arskom/spyne.git into
>   'D:\Workplace\Python Spyne'
> * have Python 2.7.6 x64 installation with setuptools 3.6
> * CPython 2.7.6 installed in 'C:\Program Files\Python\Python276'
> * just in case paths with spaces might be causing problems, I
>   linked the Spyne project location to 'Y:\' and the Python
>   installation to 'X:\'.
> * I run 'X:\python.exe setup.py test' from the 'Y:\' folder and
>   loads of packages get installed after which the tests run.
>
>   The first batch of tests runs but there the following tests fail:
>
>> spyne\test\test_util.py:197: in test_log_repr_complex
>>>       assert log_repr(val) == "Z(z=['abc', 'abc', 'abc', 'abc', 
>>> (...)])"
>> E       AssertionError: assert 'Z(z=[])' == "Z(z=['abc', 'abc', 
>> 'abc', 'abc', (...)])"
>> E         - Z(z=[])
>> E         + Z(z=['abc', 'abc', 'abc', 'abc', (...)])
>>
>> spyne\test\test_util.py:177: in test_log_repr_simple
>>>       assert log_repr(['a','b','c'], Array(String)) ==  "['a', 'b', 
>>> 'c']"
>> E       AssertionError: assert '[]' == "['a', 'b', 'c']"
>> E         - []
>> E         + ['a', 'b', 'c']
>
>   They also result in the following two exceptions:
>


Yeah, Travis thought the tests passed because of the issue you filed 
yesterday. I fixed that and I'll fix the tests asap.


As for the following issues you reported, it is mostly because you're 
running things on windows. Actually, you might be the first person ever 
running Spyne test suite on windows.

>> Exception in thread Thread-6:
>> Traceback (most recent call last):
>>   File "X:\lib\threading.py", line 810, in __bootstrap_inner
>>     self.run()
>>   File "X:\lib\threading.py", line 763, in run
>>     self.__target(*self.__args, **self.__kwargs)
>>   File "X:\lib\multiprocessing\pool.py", line 329, in _handle_workers
>>     debug('worker handler exiting')
>> TypeError: 'NoneType' object is not callable
>>
>> Exception in thread Thread-7:
>> Traceback (most recent call last):
>>   File "X:\lib\threading.py", line 810, in __bootstrap_inner
>>     self.run()
>>   File "X:\lib\threading.py", line 763, in run
>>     self.__target(*self.__args, **self.__kwargs)
>>   File "X:\lib\multiprocessing\pool.py", line 353, in _handle_tasks
>>     debug('task handler got sentinel')
>> TypeError: 'NoneType' object is not callable
>
>   After that tox failed miserably, which turned out to be caused by 
> Python installation being located in the root folder. Actually I think 
> it's virtualenv's fault but I did not do any significant debugging 
> here. Once I relocated it to 'X:\Python276' instead of 'X:\' and reran 
> the tests using 'X:\Python276\python.exe setup.py test' - tox started 
> to install django into its virtual environments and all the django 
> tests were reported as passed (and d*mn those django installations 
> take a looooong time :-D).
>

yes, but just the first time :)

>   After that I don't think your 'call_pytest_subprocess' tests work 
> quite the way you'd want them, at least not on Windows - whichever one 
> I try to run I get the following error:
>
>> Traceback (most recent call last):
>>   File "setup.py", line 438, in <module>
>>     'test_multi_python': RunMultiPythonTests
>>   File "X:\Python276\lib\distutils\core.py", line 152, in setup
>>     dist.run_commands()
>>   File "X:\Python276\lib\distutils\dist.py", line 953, in run_commands
>>     self.run_command(cmd)
>>   File "X:\Python276\lib\distutils\dist.py", line 972, in run_command
>>     cmd_obj.run()
>>   File "build\bdist.win-amd64\egg\setuptools\command\test.py", line 
>> 146, in run
>>   File "build\bdist.win-amd64\egg\setuptools\command\test.py", line 
>> 127, in with_project_on_sys_path
>>   File "setup.py", line 248, in run_tests
>>     capture=self.capture) or ret
>>   File "setup.py", line 126, in call_pytest_subprocess
>>     return call_test(pytest.main, args, tests)
>>   File "setup.py", line 64, in call_test
>>     p.start()
>>   File "X:\Python276\lib\multiprocessing\process.py", line 130, in start
>>     self._popen = Popen(self)
>>   File "X:\Python276\lib\multiprocessing\forking.py", line 277, in 
>> __init__
>>     dump(process_obj, to_child, HIGHEST_PROTOCOL)
>>   File "X:\Python276\lib\multiprocessing\forking.py", line 199, in dump
>>     ForkingPickler(file, protocol).dump(obj)
>>   File "X:\Python276\lib\pickle.py", line 224, in dump
>>     self.save(obj)
>>   File "X:\Python276\lib\pickle.py", line 331, in save
>>     self.save_reduce(obj=obj, *rv)
>>   File "X:\Python276\lib\pickle.py", line 419, in save_reduce
>>     save(state)
>>   File "X:\Python276\lib\pickle.py", line 286, in save
>>     f(self, obj) # Call unbound method with explicit self
>>   File "X:\Python276\lib\pickle.py", line 649, in save_dict
>>     self._batch_setitems(obj.iteritems())
>>   File "X:\Python276\lib\pickle.py", line 681, in _batch_setitems
>>     save(v)
>>   File "X:\Python276\lib\pickle.py", line 286, in save
>>     f(self, obj) # Call unbound method with explicit self
>>   File "X:\Python276\lib\pickle.py", line 748, in save_global
>>     (obj, module, name))
>> pickle.PicklingError: Can't pickle <function _ at 
>> 0x0000000003FDCAC8>: it's not found as __main__._
>


You might know that Windows doesn't have the fork() syscall. 
Multiprocessing relies heavily on fork() to work on unix. You know what 
it does on windows? Starts a new Python process, pickles EVERYTHING, 
sends them to the new process and tries to reconstruct the state in the 
"child" process to simulate a fork.

So to use multiprocessing in a cross platform way, you need to make sure 
EVERYTHING is pickleable. In my book, that's just insane, so I didn't 
bother with windows. If anybody's got a better idea for calling py.test 
from the setup script, I'm open to suggestions.

https://docs.python.org/2/library/multiprocessing.html#windows

>   And the main test suite seems to exit and leave behind a background 
> runner process that exits like this:
>
>> Traceback (most recent call last):
>>   File "<string>", line 1, in <module>
>>   File "X:\Python276\lib\multiprocessing\forking.py", line 380, in main
>>     prepare(preparation_data)
>>   File "X:\Python276\lib\multiprocessing\forking.py", line 495, in 
>> prepare
>>     '__parents_main__', file, path_name, etc
>>   File "Y:\setup.py", line 438, in <module>
>>     'test_multi_python': RunMultiPythonTests
>>   File "X:\Python276\lib\distutils\core.py", line 152, in setup
>>     dist.run_commands()
>>   File "X:\Python276\lib\distutils\dist.py", line 953, in run_commands
>>     self.run_command(cmd)
>>   File "X:\Python276\lib\distutils\dist.py", line 972, in run_command
>>     cmd_obj.run()
>>   File "build\bdist.win-amd64\egg\setuptools\command\test.py", line 
>> 146, in run
>>   File "build\bdist.win-amd64\egg\setuptools\command\test.py", line 
>> 127, in with_project_on_sys_path
>>   File "Y:\setup.py", line 248, in run_tests
>>     capture=self.capture) or ret
>>   File "Y:\setup.py", line 126, in call_pytest_subprocess
>>     return call_test(pytest.main, args, tests)
>>   File "Y:\setup.py", line 64, in call_test
>>     p.start()
>>   File "X:\Python276\lib\multiprocessing\process.py", line 130, in start
>>     self._popen = Popen(self)
>>   File "X:\Python276\lib\multiprocessing\forking.py", line 258, in 
>> __init__
>>     cmd = get_command_line() + [rhandle]
>>   File "X:\Python276\lib\multiprocessing\forking.py", line 358, in 
>> get_command_line
>>     is not going to be frozen to produce a Windows executable.''')
>> RuntimeError:
>>             Attempt to start a new process before the current process
>>             has finished its bootstrapping phase.
>>
>>             This probably means that you are on Windows and you have
>>             forgotten to use the proper idiom in the main module:
>>
>>                 if __name__ == '__main__':
>>                     freeze_support()
>>                     ...
>>
>>             The "freeze_support()" line can be omitted if the program
>>             is not going to be frozen to produce a Windows executable.
>
>   If you have any ideas, I'd love to get this working. :-D
>

Well, we need a call_pytest_subprocess that works on windows. Probably 
one that doesn't use multiprocessing.

You can just do py.test test_suds.py though, it should work.

>   Oh, one more test left - the final one, called using the 
> 'call_trial' function:
>
> 1. It requires the pywin32 package to be installed in the target 
> Python environment (Twisted uses the win32api module module), and 
> setup does not install that itself. And it's a bitchy module to 
> install automatically since it comes as a windows installer and can 
> not be automatically installed using pip. :-(
>
> 2. When you do install pywin32 it spurts out lots of debug & info log 
> output which I have no idea what to do with.
>
> 3. Among that ton of text the following error message gets repeated a 
> few times which I also have no idea what to do with or whether it even 
> represents a test failure or not:
>
>> ERROR:spyne.application:Possible
>> Traceback (most recent call last):
>>   File "y:\spyne\application.py", line 144, in process_request
>>     ctx.out_object = self.call_wrapper(ctx)
>>   File "y:\spyne\application.py", line 199, in call_wrapper
>>     retval = ctx.descriptor.service_class.call_wrapper(ctx)
>>   File "y:\spyne\service.py", line 207, in call_wrapper
>>     return ctx.function(*ctx.in_object)
>>   File "y:\spyne\test\interop\server\_service.py", line 336, in 
>> python_exception
>>     raise Exception("Possible")
>> Exception: Possible
>

This is output from a test to see how twisted handles unexpected 
exceptions. So this is not an error.

> 4. And similar to 3. the following error appears once:
>
>> ERROR:spyne.application:Fault(Plausible: 'A plausible fault')
>> Traceback (most recent call last):
>>   File "y:\spyne\application.py", line 144, in process_request
>>     ctx.out_object = self.call_wrapper(ctx)
>>   File "y:\spyne\application.py", line 199, in call_wrapper
>>     retval = ctx.descriptor.service_class.call_wrapper(ctx)
>>   File "y:\spyne\service.py", line 207, in call_wrapper
>>     return ctx.function(*ctx.in_object)
>>   File "y:\spyne\test\interop\server\_service.py", line 340, in 
>> soap_exception
>>     raise Fault("Plausible", "A plausible fault", 
>> 'http://faultactor.example.com')
>> Fault: Fault(Plausible: 'A plausible fault')
>

And this is testing how twisted handles "expected" exceptions. So this 
also is not an error.

Thank you for your detailed explanations. I'm adding a "running on 
windows" section to the test suite readme and update it according to 
your input.

Best regards,
Burak


More information about the people mailing list