Skip to content
Snippets Groups Projects
Notes.rst 11.2 KiB
Newer Older
* Build a PrivateStorage-related Android application
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed

  * Build an app that can interact with magic-folders somehow
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed

    * Build an app that can read magic-folders and download linked files
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed

Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
      * Build libchallenge_bypass_ristretto for armv7a and aarch64 Android

	* nixpkgs *used to* use "crates-io.nix" which was quite friendly to cross-compilation!

	  But it got thrown in the trash and now you are directed to naersk or crate2nix.

	* cross-compilation with crate2nix expressions is not straightforward

	  Does it work at all, for anything?
	  Who knows.
	  There's a ticket open.
	  I haven't yet figured out how to make it go.

	* naersk looks super easy to use!

	  But does it support cross-compilation?
	  It is not clear.
	  (Naive?) Attempts to do so on nixpkgs master@HEAD fail with infinite recursion errors.
	  On nixos-22.05 with some patches applied ... maybe it will work?
	  (As of 2022-12-05 16:04, build is still running)

	* Either way we need to preserve our postInstall step to get the .h and .pc files

      * Build an app that runs a Haskell Tahoe-LAFS implementation on Android

	* Reflex-FRP / Obsidian Systems Obelisk?
	  https://reflex-frp.org/tutorial
	  https://github.com/obsidiansystems/obelisk

	  * The Obelisk readme basically works.
	  * Adding a native extension seems tricky.

	    * Obelisk pins a reflex-platform
	    * reflex-platform pins a nixpkgs
	    * reflex-platform various arguments to control nixpkgs

	      * nixpkgsFunc to completely replace the nixpkgs used
	      * nixpkgsOverlays to contribute overlays to nixpkgs
	      * config to pass other config to nixpkgs

	    * Obelisk does not accept nixpkgsFunc or nixpkgsOverlays

	      * But it does accept reflex-platform-func
	      * So override this with a version that passes an overlay to reflex, I guess



      * Build an app that runs the Python Tahoe-LAFS implementation on Android
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed

	* Build Tahoe-LAFS & ZKAPAuthorizer for Android
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed

Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
          * Build Python for Android
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
            * Terms
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed

Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
              * "Build system" is the system where you compile a package.
              * "Host system" is the system where you run the package.
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
            * Try using nix (nixpkgs 3d58e3a31ee55554fe26650bb59db7d6d0e97625 / https://github.com/NixOS/nixpkgs/pull/201734 rebased onto staging) ::
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                nix build .#pkgsCross.aarch64-android.python3Minimal
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
            * Or, Chaquopy has a build
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
          * Build TahoeLAFS/ZKAPAuthorizer dependencies for Android
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
            * zfec
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
            * ::
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                nix build .#pkgsCross.aarch64-android.python3Minimal.pkgs.zfec
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
              However, this fails.
              Building an aarch64-android wheel package fails::

                  running bdist_wheel
                  Traceback (most recent call last):
                    File "/build/wheel-0.37.1-source/nix_run_setup", line 8, in <module>
                      exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
                    File "setup.py", line 4, in <module>
                      setup(maintainer=u'Alex Grönholm')
                    File "/nix/store/87qbq45psc97az9h5qmn6csfvrj9gl6g-python3.10-bootstrapped-pip-22.2.2/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
                      return distutils.core.setup(**attrs)
                  .
                  .
                  .
                    File "/nix/store/lygssilaq8p1a85pmbnfd7k2yw76kf2k-python3-minimal-3.10.8/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
                      module = import_module(match.group('module'))
                    File "/nix/store/lygssilaq8p1a85pmbnfd7k2yw76kf2k-python3-minimal-3.10.8/lib/python3.10/importlib/__init__.py", line 126, in import_module
                      return _bootstrap._gcd_import(name[level:], package, level)
                  .
                  .
                  .
                    File "/nix/store/87qbq45psc97az9h5qmn6csfvrj9gl6g-python3.10-bootstrapped-pip-22.2.2/lib/python3.10/site-packages/setuptools/command/build_ext.py", line 26, in <module>
                      get_config_var("LDSHARED")
                    File "/nix/store/87qbq45psc97az9h5qmn6csfvrj9gl6g-python3.10-bootstrapped-pip-22.2.2/lib/python3.10/site-packages/setuptools/_distutils/sysconfig.py", line 574, in get_config_var
                      return get_config_vars().get(name)
                    File "/nix/store/87qbq45psc97az9h5qmn6csfvrj9gl6g-python3.10-bootstrapped-pip-22.2.2/lib/python3.10/site-packages/setuptools/_distutils/sysconfig.py", line 553, in get_config_vars
                      _config_vars = sysconfig.get_config_vars().copy()
                    File "/nix/store/lygssilaq8p1a85pmbnfd7k2yw76kf2k-python3-minimal-3.10.8/lib/python3.10/sysconfig.py", line 615, in get_config_vars
                      _init_posix(_CONFIG_VARS)
                    File "/nix/store/lygssilaq8p1a85pmbnfd7k2yw76kf2k-python3-minimal-3.10.8/lib/python3.10/sysconfig.py", line 477, in _init_posix
                      _temp = __import__(name, globals(), locals(), ['build_time_vars'], 0)
                  ModuleNotFoundError: No module named '_sysconfigdata__linux_aarch64-linux-android'

Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
              ``_sysconfigdata__linux_aarch64-linux-android`` is meant to hold information about the host platform for the software to be built.
              CPython doesn't have a good cross-compilation system.
              It will guess a sysconfidata module name to import or use a value given by _PYTHON_SYSCONFIGDATA_NAME.
              nixpkgs defines a value for _PYTHON_SYSCONFIGDATA_NAME.
              In this case it sets it to ``_sysconfigdata__linux_aarch64-linux-android``.
              The CPython that is running is the "Python for build" interpreter -
              the interpreter that is able to run on the build system.
              It is searching for the sysconfigdata for the "Python for host" interpreter -
              the interpreter that is able to run on the system where the package being built will be used.
              The "Python for build" interpreter has ``site-packages/_sysconfigdata__linux_x86_64-linux-gnu.py``
              which is a symlink to ``_sysconfigdata__linux_x86_64-linux-gnu.py`
              which exists
              and contains information about the build system.
              The pkgsCross.aarch64-android.python3Minimal.pkgs.zfec derivation has PYTHONPATH set in its environment.
              The value (split up for readability) is

              * /nix/store/53afd480zidx2p9j9c7dsgckjgfk66h1-python3-minimal-aarch64-unknown-linux-android-3.10.8/lib/python3.10/site-packages
              * /nix/store/azdg3l16q2d03g8drgr94bg1069mfzpq-python3.10-bootstrapped-pip-22.2.2/lib/python3.10/site-packages
              * /nix/store/spk2md1jgwrg6id10j1rl9448s4naa94-python3.10-setuptools-65.3.0/lib/python3.10/site-packages
              * /nix/store/537cvh8mqdaglc0an8ndmzszllh1lwac-python3-minimal-3.10.8/lib/python3.10/site-packages

              The first entry is a directory which contains ``_sysconfigdata__linux_.py``
              which is a symlink to ``../_sysconfigdata__linux_.py``
              which exists
              and contains information about the host system.

              The ``_sysconfigdata_*`` file is

              * named by ``sysconfig._get_sysconfigdata_name``
              * and written by ``sysconfig._generate_posix_vars``
              * which is called by ``sysconfig._main``
              * which is invoked by the ``pybuilddir.txt`` Makefile target
              * which is depended on by the ``sharedmods`` Makefile target
              * which is depended on by the ``build_all`` and ``sharedinstall`` Makefile targets
              * ``build_all`` is the default Make target
              * nixpkgs python3Minimal expression defines a postFixup hook
                * ``sysconfigdataHook`` ::

                    sysconfigdataHook() {
                      if [ "$1" = '${placeholder "out"}' ]; then
                        export _PYTHON_HOST_PLATFORM='${pythonHostPlatform}'
                        export _PYTHON_SYSCONFIGDATA_NAME='${pythonSysconfigdataName}'
                      fi
                    }

                    addEnvHooks "$hostOffset" sysconfigdataHook

Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                * In postFixup (after build, before install) sysconfigdataHook is added as a $hostOffset env hook.
                  This means ... ??????????????????????
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                * Some observations
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                  * On a native build
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                    * buildPhase started
                    * lots of core compiling
                    * ``LD_LIBRARY_PATH=/build/Python-3.10.8 ./python -E -S -m sysconfig --generate-posix-vars``
                    * lots of extension module compiling
                    * ``_sysconfigdata__linux_x86_64-linux-gnu.py`` generated/written
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                  * On an aarch64-multiplatform build
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                    * buildPhase started
                    * lots of core compiling
                    * ``_PYTHON_PROJECT_BASE=/build/Python-3.10.8 _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_aarch64-linux-gnu python -S -m sysconfig --generate-posix-vars``
                    * lots of extension module compiling
                    * ``_sysconfigdata__linux_aarch64-linux-gnu.py'`` generated/written
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                  * On an aarch64-android build
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                    * buildPhase started
                    * lots of core compiling
                    * ``_PYTHON_PROJECT_BASE=/build/Python-3.10.8 _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_ python -S -m sysconfig --generate-posix-vars``
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                      * Uh oh.
                        ``_sysconfigdata__linux_``
                        Why?
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                        * We expect to see ``_sysconfigdata__linux_aarch64-linux-android``
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                          * This means the "abi flags" are ""
                            The "platform" is "linux" (questionable...)
                            The "multiarch" is "aarch64-linux-android" (is that coherent?)
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                          * The exact string might not really matter, we just need it to agree.
                            Where is the multiarch suffix going?
                            Or who is setting this unrelated value?
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
                          * overrideAttrs ``NIX_DEBUG = "6"``; into the derivation for extremely verbose output ...

                      * The nixpkgs expression does not result in this string
                        (by inspection with builtins.trace)

              * python-cryptography
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed

		* cffi-based projects seem to be tricky!

Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
              * python-challenge-bypass-ristretto

	    * crossenv can do it!  Mostly!
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
	      See crossenv subdirectory.

	      These work:

		* python-challenge-bypass-ristretto

		  Note that you at least need to install cffi in the
		  build-python environment.  Maybe some other deps require
		  this as well..

		* bcrypt
		* cbor2
		* netifaces
		* pycddl
		* zope.interface
		* pyyaml

	      These fail:
		  configure: error: cannot run C compiled programs.
		  If you meant to cross compile, use `--host'.
		  ld: error: unable to find library -lpthread
Jean-Paul Calderone's avatar
Jean-Paul Calderone committed
          * Integrate a Python Tahoe-LAFS runtime with a GUI

            * Build a GUI