Python modules not found even after specifying python_userpath

[Michael Harris differentiated]

I am having some trouble getting the new Python integration to work in Stata 16. Specifically, some of my modules are not found even after I specify the python executable file and set the python userpath.

I am on a linux machine on which I do not have administrative privileges but do have substantial freedom to keep files and install software locally in my home directory. The Stata license is system-wide but my Python 3.7 installation is local (via miniconda3). There are other Python versions installed in system locations but I do not want to use them, as I would like to be able to manage my own Python packages which are not all available on the wider system.

I have instructed Stata to find my local Python executable by setting python_exec, and have also set the python_userpath variable to the site-packages/ directory of my local Python installation, with the prepend option. The directory structure is such that, within site-packages/, there are many directories corresponding with package names and these contain the .py files for the installed packages. It is a standard miniconda3 structure. Running

python query

confirms that the two variables have been set and that the correct Python system has been identified with its corresponding library.

Some packages are found; for example,

python where numpy

displays the correct location for numpy. However,

python where pandas

results in error r(601), "Python module pandas not found," even though pandas is installed and its file structure is extremely similar to that of numpy (both relevant files are named __init__.py and are in an eponymous directory within site-packages/).

I tried adding site-packages/pandas/ to the library path, but with no success. I also tried creating a symbolic link to pandas/__init__.py with the name pandas.py in the site-packages/ directory, hoping that Stata would be looking for a file with that name in that location, but pandas is still not found. I'm not sure what might be causing this and I feel I have exhausted the available documentation. Please advise.

[Hua Peng (StataCorp)]

Could you show us the output of

python query
python which numpy

[Michael Harris differentiated]

. python query
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    Python Settings
      set python_exec      /path/to/home/username/local/miniconda3/bin/python
      set python_userpath  /path/to/home/username/local/miniconda3/lib/python3.7/site-packages

    Python system information
      initialized          yes
      version              3.7.3
      architecture         64-bit
      library path         /path/to/home/username/local/miniconda3/lib/libpython3.7m.so

. python which numpy
<module 'numpy' from '/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/numpy/__init__.py'>

. python which pandas
Python module pandas not found
r(601);

. ls /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas

total 80
drwxr-xr-x  5 username users  4096 Jul 16 07:10 api/
drwxr-xr-x  3 username users  4096 Jul 16 07:10 arrays/
drwxr-xr-x  4 username users  4096 Jul 16 07:10 compat/
-rw-r--r--  1 username users 18042 Jul 16 07:10 conftest.py
drwxr-xr-x 13 username users  4096 Jul 16 07:10 core/
drwxr-xr-x  3 username users  4096 Jul 16 07:10 errors/
-rw-r--r--  1 username users  3973 Jul 16 07:10 __init__.py
drwxr-xr-x  8 username users  4096 Jul 16 07:10 io/
drwxr-xr-x  4 username users  4096 Jul 16 07:10 _libs/
drwxr-xr-x  3 username users  4096 Jul 16 07:10 plotting/
drwxr-xr-x  2 username users  4096 Jul 16 16:40 __pycache__/
-rw-r--r--  1 username users   158 Jul 16 07:10 testing.py
drwxr-xr-x 27 username users  4096 Jul 16 07:10 tests/
drwxr-xr-x  3 username users  4096 Jul 16 07:10 tseries/
drwxr-xr-x  3 username users  4096 Jul 16 07:10 util/
-rw-r--r--  1 username users   546 Jul 16 07:10 _version.py

and I get the same error if I change the python_userpath to also include /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas.

pandas works properly from the shell like

!
python
import pandas
...

but here's what happens when I try the same thing more directly in Stata:

. python
----------------------------------------------- python (type end to exit) ----------------------------------------------------------------------------------------------
>>> import pandas
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/__init__.py", line 42, in <module>
    from pandas.core.api import *
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/core/api.py", line 26, in <module>
    from pandas.core.groupby import Grouper
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/core/groupby/__init__.py", line 1, in <module>
    from pandas.core.groupby.groupby import GroupBy  # noqa: F401
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/core/groupby/groupby.py", line 37, in <module>
    from pandas.core.frame import DataFrame
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/core/frame.py", line 100, in <module>
    from pandas.core.series import Series
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/core/series.py", line 4390, in <module>
    Series._add_series_or_dataframe_operations()
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 10138, in _add_series_or_dataframe_operations
    from pandas.core import window as rwindow
  File "/path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/core/window.py", line 14, in <module>
    import pandas._libs.window as libwindow
ImportError: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_lib
> s/window.cpython-37m-x86_64-linux-gnu.so)
r(7102);
>>>

I am wondering whether the mention of /lib64/libstdc++.so.6 is a clue that Stata is looking in system files rather than the expected behavior of looking in my local files; otherwise, I don't have an idea of what the issue might be. I do have libgcc installed locally, and again, the shell does not give me this problem.

[Hua Peng (StataCorp)]

Thanks for the output, we are looking into the issue and will report back when the problem is identified.

[Pete Huckelba (StataCorp)]

This sounds like you may have a dated operating system that does not have a current enough version of libstdc++. Pandas gets around this by using libstdc++ that is packaged by miniconda, namely for the library /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_libs/window.cpython-37m-x86_64-linux-gnu.so. On the other hand, Stata is able to use the system libstdc++ without issue but both libraries can not be loaded at the same time.

Do you know what the Linux distribution and version level are? Also providing the output from command line for the following commands may provide some clarity:

uname -a
ls -la /lib64/libstdc++*
strings /lib64/libstdc++.so.6 | grep GLIBCXX
strings /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_libs/window.cpython-37m-x86_64-linux-gnu.so | grep GLIBCXX
ldd /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_libs/window.cpython-37m-x86_64-linux-gnu.so
ldd /path/to/stata16/xstata

-Pete

[Michael Harris differentiated]

I'm on Red Hat Enterprise Linux Server 7.6. If the system files are outdated there's nothing I can do, but I can get local versions of libraries installed with a mix of yumdownloader, rpm2cpio, and cpio commands. Perhaps Stata will not find those, but it would be nice if I could point Stata to them.

Here's the requested output (the particular node I am using varies as assigned by slurm, but as far as I understand all nodes have access to the same system files):

+ uname -a
Linux node.do.ma.in 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
+ ls -la /lib64/libstdc++.so.5 /lib64/libstdc++.so.5.0.7 /lib64/libstdc++.so.6 /lib64/libstdc++.so.6.0.19 | grep GLIBCXX
lrwxrwxrwx 1 root root     18 Oct 28  2016 /lib64/libstdc++.so.5 -> libstdc++.so.5.0.7
-rwxr-xr-x 1 root root 830776 Nov 13  2014 /lib64/libstdc++.so.5.0.7
lrwxrwxrwx 1 root root     19 Apr  1 08:08 /lib64/libstdc++.so.6 -> libstdc++.so.6.0.19
-rwxr-xr-x 1 root root 991616 Jan 17 17:02 /lib64/libstdc++.so.6.0.19
+ strings /lib64/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_3.4.14
GLIBCXX_3.4.15
GLIBCXX_3.4.16
GLIBCXX_3.4.17
GLIBCXX_3.4.18
GLIBCXX_3.4.19
GLIBCXX_DEBUG_MESSAGE_LENGTH
+ strings /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_libs/window.cpython-37m-x86_64-linux-gnu.so
GLIBCXX_3.4.21
GLIBCXX_3.4
_ZSt20__throw_length_errorPKc@@GLIBCXX_3.4
_ZdlPv@@GLIBCXX_3.4
_ZTISt8bad_cast@@GLIBCXX_3.4
_ZTISt12out_of_range@@GLIBCXX_3.4
_ZTISt15underflow_error@@GLIBCXX_3.4
_ZTISt12domain_error@@GLIBCXX_3.4
_ZTISt9exception@@GLIBCXX_3.4
_ZTINSt8ios_base7failureB5cxx11E@@GLIBCXX_3.4.21
_ZTISt14overflow_error@@GLIBCXX_3.4
_ZTISt10bad_typeid@@GLIBCXX_3.4
_ZSt17__throw_bad_allocv@@GLIBCXX_3.4
_ZTISt11range_error@@GLIBCXX_3.4
_Znwm@@GLIBCXX_3.4
_ZTISt16invalid_argument@@GLIBCXX_3.4
_ZTISt9bad_alloc@@GLIBCXX_3.4
+ ldd /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_libs/window.cpython-37m-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffe7fdc0000)
    libstdc++.so.6 => /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_libs/../../../../libstdc++.so.6 (0x00007f50ac80e000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f50ac50c000)
    libgcc_s.so.1 => /path/to/home/username/local/miniconda3/lib/python3.7/site-packages/pandas/_libs/../../../../libgcc_s.so.1 (0x00007f50acab5000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f50ac2f0000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f50abf23000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f50ac982000)
+ ldd /path/to/apps/stata/16/xstata
    linux-vdso.so.1 =>  (0x00007ffcfb7c6000)
    libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f37c5581000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f37c536b000)
    libgtk-x11-2.0.so.0 => /lib64/libgtk-x11-2.0.so.0 (0x00007f37c4ce9000)
    libgdk-x11-2.0.so.0 => /lib64/libgdk-x11-2.0.so.0 (0x00007f37c4a28000)
    libatk-1.0.so.0 => /lib64/libatk-1.0.so.0 (0x00007f37c4802000)
    libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007f37c4463000)
    libpangoft2-1.0.so.0 => /lib64/libpangoft2-1.0.so.0 (0x00007f37c424d000)
    libpangocairo-1.0.so.0 => /lib64/libpangocairo-1.0.so.0 (0x00007f37c403f000)
    libgdk_pixbuf-2.0.so.0 => /lib64/libgdk_pixbuf-2.0.so.0 (0x00007f37c3e17000)
    libcairo.so.2 => /lib64/libcairo.so.2 (0x00007f37c3ae0000)
    libpango-1.0.so.0 => /lib64/libpango-1.0.so.0 (0x00007f37c389a000)
    libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f37c35db000)
    libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f37c3399000)
    libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007f37c3149000)
    libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f37c2f45000)
    libgthread-2.0.so.0 => /lib64/libgthread-2.0.so.0 (0x00007f37c2d43000)
    librt.so.1 => /lib64/librt.so.1 (0x00007f37c2b3b000)
    libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f37c2825000)
    libpng12.so.0 => /lib64/libpng12.so.0 (0x00007f37c25fe000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f37c23e8000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f37c21cc000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f37c1dff000)
    libm.so.6 => /lib64/libm.so.6 (0x00007f37c1afd000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f37c18f9000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f37c5888000)
    libX11.so.6 => /lib64/libX11.so.6 (0x00007f37c15bb000)
    libXfixes.so.3 => /lib64/libXfixes.so.3 (0x00007f37c13b5000)
    libXrender.so.1 => /lib64/libXrender.so.1 (0x00007f37c11aa000)
    libXinerama.so.1 => /lib64/libXinerama.so.1 (0x00007f37c0fa7000)
    libXi.so.6 => /lib64/libXi.so.6 (0x00007f37c0d97000)
    libXrandr.so.2 => /lib64/libXrandr.so.2 (0x00007f37c0b8c000)
    libXcursor.so.1 => /lib64/libXcursor.so.1 (0x00007f37c0981000)
    libXcomposite.so.1 => /lib64/libXcomposite.so.1 (0x00007f37c077e000)
    libXdamage.so.1 => /lib64/libXdamage.so.1 (0x00007f37c057b000)
    libXext.so.6 => /lib64/libXext.so.6 (0x00007f37c0369000)
    libffi.so.6 => /lib64/libffi.so.6 (0x00007f37c0161000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f37bfeff000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f37bfcd8000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f37bfabf000)
    libmount.so.1 => /lib64/libmount.so.1 (0x00007f37bf87c000)
    libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007f37bf5df000)
    libpng15.so.15 => /lib64/libpng15.so.15 (0x00007f37bf3b4000)
    libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007f37bf10b000)
    libEGL.so.1 => /lib64/libEGL.so.1 (0x00007f37beef7000)
    libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007f37becf3000)
    libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f37beacb000)
    libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007f37be8bd000)
    libGL.so.1 => /lib64/libGL.so.1 (0x00007f37be631000)
    libthai.so.0 => /lib64/libthai.so.0 (0x00007f37be425000)
    libfribidi.so.0 => /lib64/libfribidi.so.0 (0x00007f37be209000)
    libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f37bdff9000)
    libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f37bddcf000)
    libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f37bdbca000)
    libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f37bd98a000)
    libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007f37bd75c000)
    libGLdispatch.so.0 => /lib64/libGLdispatch.so.0 (0x00007f37bd4a6000)
    libXau.so.6 => /lib64/libXau.so.6 (0x00007f37bd2a2000)
    libGLX.so.0 => /lib64/libGLX.so.0 (0x00007f37bd070000)

[Pete Huckelba (StataCorp)]

Michael,

RHEL 7.6 is up to date so no need to worry about getting new libraries. You can instruct Stata to use the Miniconda libraries (which are just slightly newer) and all should be fine:

LD_LIBRARY_PATH=/path/to/home/username/local/miniconda3/lib /path/to/stata16/stata

On my CentOS 7.6 box this worked:

. python query
--------------------------------------------------------------------------------
Python Settings
set python_exec /home/cph/miniconda3/bin/python
set python_userpath /home/cph/miniconda3/lib/python3.7/site-packages

Python system information
initialized yes
version 3.7.3
architecture 64-bit
library path /home/cph/miniconda3/lib/libpython3.7m.so

. python which numpy
<module 'numpy' from '/home/cph/miniconda3/lib/python3.7/site-packages/numpy/__i
> nit__.py'>

. python which pandas
<module 'pandas' from '/home/cph/miniconda3/lib/python3.7/site-packages/pandas/_
> _init__.py'>

. python
----------------------------------------------- python (type end to exit) ------
>>> import pandas
>>> end
--------------------------------------------------------------------------------

-Pete

[Michael Harris differentiated]

This worked. So the issue was that I had neglected to make sure the Miniconda libraries got added to my LD_LIBRARY_PATH in linux. This would have also given me issues with other software later on, so thanks for helping me realize that.