macOS cython warnings in osx

[thekev]

Has anyone used cython in OSX? If so do you know a reasonable way to deal with these warnings? I would prefer not to suppress legitimate ones. It generates types for all python types if I'm not mistaken, as those resemble the python C api. I'm not sure this is actually normal though. The warning came up both with both clang and gcc (installed via brew and set via CC and CXX).

[1/1] Cythonizing blah.pyx
running build_ext
building 'blah' extension
/usr/bin/gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/include/python3.5m -c blah.c -o build/temp.macosx-10.11-x86_64-3.5/blah.o
blah.c:1670:28: warning: unused function '__Pyx_PyObject_AsString' [-Wunused-function]
static CYTHON_INLINE char* __Pyx_PyObject_AsString(PyObject* o) {
                           ^
blah.c:1667:32: warning: unused function '__Pyx_PyUnicode_FromString' [-Wunused-function]
static CYTHON_INLINE PyObject* __Pyx_PyUnicode_FromString(const char* c_str) {
                               ^
blah.c:1732:26: warning: unused function '__Pyx_PyObject_IsTrue' [-Wunused-function]
static CYTHON_INLINE int __Pyx_PyObject_IsTrue(PyObject* x) {
                         ^
blah.c:1782:33: warning: unused function '__Pyx_PyIndex_AsSsize_t' [-Wunused-function]
static CYTHON_INLINE Py_ssize_t __Pyx_PyIndex_AsSsize_t(PyObject* b) {
                                ^
blah.c:1844:33: warning: unused function '__Pyx_PyInt_FromSize_t' [-Wunused-function]
static CYTHON_INLINE PyObject * __Pyx_PyInt_FromSize_t(size_t ival) {
                                ^
blah.c:1202:32: warning: unused function '__Pyx_PyInt_From_long' [-Wunused-function]
static CYTHON_INLINE PyObject* __Pyx_PyInt_From_long(long value) {
                               ^
blah.c:1251:27: warning: function '__Pyx_PyInt_As_long' is not needed and will not be emitted
      [-Wunneeded-internal-declaration]
static CYTHON_INLINE long __Pyx_PyInt_As_long(PyObject *x) {
                          ^
blah.c:1436:26: warning: function '__Pyx_PyInt_As_int' is not needed and will not be emitted
      [-Wunneeded-internal-declaration]
static CYTHON_INLINE int __Pyx_PyInt_As_int(PyObject *x) {
                         ^
8 warnings generated.
/usr/bin/gcc -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk build/temp.macosx-10.11-x86_64-3.5/blah.o -L/usr/local/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/sqlite/lib -o

 

[chown33]

I know almost nothing about cython, but I've had to deal with recalcitrant compiler options before.

I think you're going to need to disable these two warnings:

-Wunused-function
-Wunneeded-internal-declaration​

Since they aren't listed explicitly in the 'gcc' command shown, my first guess is they're implied by -Wall. That's pretty much what I had to deal with before: disabling warnings enabled by -Wall.

The standard way of disabling warnings is to prefix the name with "no-", so:

-Wno-unused-function
-Wno-unneeded-internal-declaration​

I assume you have enough control over the build process to add those. If not, they might be passable in the CFLAGS environment variable (if build process uses 'make').


As I understand Cython, you write source in Python and it's then compiled down to C. If that's accurate, then simply disabling the warnings in the command line should solve the problem completely.

Furthermore, disabling the specific warnings shouldn't lead to any missed warnings (true positives, as opposed to the false positives you're seeing now). Right now, all the warnings you're seeing appear to be as a result of the "cythonizing" of the Python-to-C code-generator. In other words, the code-generator is being safe and comprehensive in emitting code, even if it emits unused code (funcs or decls).

As long as the source is just Python, the generated code can be assumed to be correct and complete, even if it contains code that isn't used. If that assumption were wrong (code is incorrect or incomplete), then the code-generation itself is broken, not the 'gcc' compiler or its warnings.

However, if you can write source in C within the .pyx file, then you might still be able to have some true positive warnings. For example, if you defined a C static function that wasn't used, that might be worth warning about. On the other hand, maybe it isn't: if you have a C static function that isn't used, how much do you care? I'd only care if it was unused in error, and there are other means for dealing with that (unit tests, e.g.).


When I dealt with this issue of unwanted warnings before, it was after an upgrade to 'gcc' where more warnings were enabled by -Wall, some of which didn't previously exist, so the developer couldn't have disabled them. That is, older code got compiled by a newer and pickier compiler, so we had to go back in and disable some of the compiler's new-found pickiness.

 

[thekev]

I know almost nothing about cython, but I've had to deal with recalcitrant compiler options before.

I think you're going to need to disable these two warnings:

-Wunused-function
-Wunneeded-internal-declaration​

Since they aren't listed explicitly in the 'gcc' command shown, my first guess is they're implied by -Wall. That's pretty much what I had to deal with before: disabling warnings enabled by -Wall.

The standard way of disabling warnings is to prefix the name with "no-", so:

-Wno-unused-function
-Wno-unneeded-internal-declaration​

I assume you have enough control over the build process to add those. If not, they might be passable in the CFLAGS environment variable (if build process uses 'make').

It's using distutils, which doesn't appear to call any variation of make. It's a much older option. I may look into the amount of effort required to deal with both the C and numpy apis, but this is usually considered faster. Here's my equivalent to hello world.

from distutils.core import setup
from Cython.Build import cythonize
import os;
os.environ["CC"] = "/usr/bin/gcc"
os.environ["CXX"] = "/usr/bin/gcc"

setup(
    name = 'blah',
    ext_modules = cythonize('blah.pyx'),
)

I've used it on small snippets of code in the past. I tried gcc because I thought clang might be the problem. Under Linux, gcc is the standard compiler for cython code.

As I understand Cython, you write source in Python and it's then compiled down to C. If that's accurate, then simply disabling the warnings in the command line should solve the problem completely.

Cython can also wrap C code. Without it, I would have to deal with both C and numpy apis. It's numerical code, and python lists aren't really ideal. At the same time I can't write all of the tight loops to conform with numpy's broadcasting rules, so a for loop would incur interpreter overhead at each iteration. Some of the other options like numba's jit compiler aren't really sufficient here.

However, if you can write source in C within the .pyx file, then you might still be able to have some true positive warnings. For example, if you defined a C static function that wasn't used, that might be worth warning about. On the other hand, maybe it isn't: if you have a C static function that isn't used, how much do you care? I'd only care if it was unused in error, and there are other means for dealing with that (unit tests, e.g.).

That's my reason for caring. I'll write some unit tests as the code progresses. You are right though. It's generated by -Wall, which is a default option here. distutils is annoying, but cython still uses it. I think there are other options, but I haven't tried them.

When I dealt with this issue of unwanted warnings before, it was after an upgrade to 'gcc' where more warnings were enabled by -Wall, some of which didn't previously exist, so the developer couldn't have disabled them. That is, older code got compiled by a newer and pickier compiler, so we had to go back in and disable some of the compiler's new-found pickiness.

Yeah I encountered a lot of OSX problems when el capitan was released, as Apple changed permissions on certain things. I encounter this issue in both gcc and clang.

 

[chown33]

The following fragment gave me an idea:

os.environ["CC"] = "/usr/bin/gcc"
os.environ["CXX"] = "/usr/bin/gcc"

If that's defining your "compiler", and that compiler is then passed a bunch of args by distutils, then you can create your own compiler and tell distutils to use that.

First, "create your own compiler" means making a shell script located in some convenient place, such as /Users/youracct/bin. Here's a simple starting point:

#!/bin/bash
/usr/bin/gcc "$@"

Next, add the -Wno-xxx options you want to the gcc line in your "compiler" file.

Then, set the a+x mode on your compiler file:

chmod a+x ~/bin/kev-compiler

Finally:

os.environ["CC"] = "/Users/youracct/bin/kev-compiler"
os.environ["CXX"] = "/Users/youracct/bin/kev-compiler"

Doing it this way means you can play around with different gcc or clang options simply by editing your "compiler" file, without having to change anything in the distutils files.

 

[thekev]

Doing it this way means you can play around with different gcc or clang options simply by editing your "compiler" file, without having to change anything in the distutils files.

Thanks. That's a really good idea. Distutils is older and basically outdated. I don't think cython will stick with it forever. As I mentioned I'm using it to wrap a combination of C and numpy data types. That otherwise requires dealing with both the C and numpy APIs, which isn't entirely straightforward. I think there are some details in the cython book on alternate workflows, but they're a bit sparse. I'll have to re-read them.