diff --git a/Notes.rst b/Notes.rst index 5189eab45bf1208e7d307d168e04781ba9bb119c..6129cd102fe063d275f4cfb715c2aa1166542754 100644 --- a/Notes.rst +++ b/Notes.rst @@ -1,3 +1,6 @@ +notes +===== + * Build a PrivateStorage-related Android application * Build an app that can interact with magic-folders somehow @@ -208,20 +211,34 @@ * python-challenge-bypass-ristretto - * crossenv can do it! + * crossenv can do it! Mostly! See crossenv subdirectory. - * However, crossenv can't do python-challenge-bypass-ristretto - or python-cryptography. + These work: + + * python-challenge-bypass-ristretto + + Note that you at least need to install cffi in the + build-python environment. Maybe some other deps require + this as well.. + + * bcrypt + * cbor2 + * netifaces + * pycddl + * zope.interface + * pyyaml + + These fail: - These seem to have cffi as a build-time dependency. - Cross-env gives the build pip env the host cffi .so so it fails. + * pynacl - Ah, but you can clobber this with:: + configure: error: cannot run C compiled programs. + If you meant to cross compile, use `--host'. - build-pip install cffi + * cryptography - And somehow this gets used instead. + ld: error: unable to find library -lpthread * Integrate a Python Tahoe-LAFS runtime with a GUI diff --git a/Summary.md b/Summary.md new file mode 100644 index 0000000000000000000000000000000000000000..6d13ef008cf492aa38939a1826e5db1fc188ff8f --- /dev/null +++ b/Summary.md @@ -0,0 +1,170 @@ +# Summary + +We have explored a couple of tools for Python Android development. +Overall the tools appear to fall far short of what is necessary to achieve our goals in a timely manner. +Additionally, the solutions that must be created to use these tools are ultimately of low and limited-term value. + +We have also explored a couple approaches for Haskell Android development. +These are also not mainstream, highly-polished tools. +However, what they accomplish they do so with less complexity than the equivalent Python tools. +Building on these tools also creates solutions that are potentially of high and long-term value. + +# The Challenge + +To build a Tahoe-LAFS-based mobile application there are N challenges to overcome: + +1. Create a user interface that allows users to interact with the system +2. Interact with Tahoe-LAFS using its cryptographic and network protocols +3. Build and distribute the entire client system + +For most of our investigation we took as an assumption that (1) would be solved by using standard mobile system UI development tools. +For example, native Android GUI functionality is readily available to applications written in Java or Kotlin. +Thus, we focused on (2) and (3). + +# Python Tools + +The motivation behind investigating Python is that it almost entirely solves challenge (2). +There is an existing Python-based Tahoe-LAFS implementation including all of the pieces we need. +Given this, we investigated how to use Python and overcome challenge (3). + +Challenge (3) can be further broken down into at least two parts: + +1. Build native library dependencies +2. Build CPython extension modules exposing those native library dependencies to Python + +We looked at five approaches for this: + +1. Kivy +2. Chaquopy +3. crossenv +4. nix +5. direct builds + +## Kivy + +Kivy apparently works for pure-Python applications. +However it does not attempts to support building Python extension modules for Android. +Tahoe-LAFS has several such dependencies, including: + +* zfec +* python-cryptography +* python-challenge-bypass-ristretto + +Therefore, +though Kivy is the most popular and well-known of these tools, +it does not actually supply any solution to challenge (3). +Additionally, +Kivy uses a fully custom build tool-chain for creating Android packages (APKs). +This adds substantially to the complexity of the system +This translated into spending much longer trying to learn how Kivy builds work and whether it is possible to customize them as necessary for our purposes. + +## Chaquopy + +Our experience with Chaquopy was similar. +Though it does not use a custom build system, +it is still not capable of building the Python extension modules we require. +The Chaquopy creator/maintainer has built several Python extension modules and makes these builds available. +However, +because Chaquopy itself does not include the tools for performing these builds, +several problems remain: + +* The modules supply by the creator/maintainer are increasingly out of date. +* Some dependencies of Tahoe-LAFS are also missing from this set. +* Finally, + this path implies using non-reproducible binary artifacts from a relatively unknown source. + We feel this is an untenable premise for a security- and privacy-focused application. + +## crossenv + +crossenv proposes *only* to solve the cross-compilation problem. +This means it would be a complement to Kivy or Chaquopy. + +Our efforts with crossenv quickly yielded success for *most* dependencies of Tahoe-LAFS. +The following modules cross-compiled (though we did not test the results): + +* python-challenge-bypass-ristretto + + Note that you at least need to install cffi in the + build-python environment. Maybe some other deps require + this as well.. + +* bcrypt +* cbor2 +* netifaces +* pycddl +* zope.interface +* pyyaml + +However, the following modules failed to cross-compile: + +* pynacl + + configure: error: cannot run C compiled programs. + If you meant to cross compile, use `--host'. + +* cryptography + + ld: error: unable to find library -lpthread + +These are substantial dependencies and it is not feasible to use Tahoe-LAFS without them. +The cross-compilation failures come from somewhere within the CPython system for building extension modules. +We spent significant time trying to understand this system +(for crossenv but also in general as part of this spike effort). +Our conclusion is that CPython is ill-suited to supporting cross-compilation and fixing these build failures might amount to doing significant implementation work on CPython itself. +Given our other attempts to integrate CPython development work upstream, +we have limited hope of successfully integrating fixes to the cross-compilation system. + +## nix / nixpkgs + +nixpkgs itself has robust support for cross-compilation as a core feature. +It appears as though someone probably used this to support our use-case at some point in time. +However, it also appears as though this feature has "bitrotted". +We were able to build many C library dependencies for Android. +However, all of the Python extension modules fail to build due to issues with Python's cross-compilation support. + +## direct builds + +This approach involved carefully hand-crafting cross-compilation instructions for individual build dependencies. +Because of the cost in effort of this approach we only tried to apply it to one of the simplest extension modules we require: zfec. + +We were eventually about to build the zfec C and Python modules for Android. +We did not test any of the build outputs. +The build setup is very fragile and requires a number of careful fix-ups at the end. +It also took several days to complete. +Most of this setup is unique to zfec and cannot be generalized to the other dependencies we need to build. +Additionall,y other dependencies themselves have more dependencies. +For example, python-cryptography depends on the entire Rust toolchain. +We anticipate applying this approach to other dependencies would be prohibitive in the amount of time required. + +# Haskell Tools + +The motivation behind investigating Haskell is that it represents a technology choice that will be valuable in the long term. +The aspects which this conclusion are based on are: + +a. Its expressive type system makes developing complex software less complicated. +b. Its runtime CPU efficiency is comparable to that of JVM-based software + (the typical runtime for Android applications). +c. We are aware that some people are using Haskell for Android apps in the real world. + +Considering the difficulty we found in using Python to address challenge (3) we find item (c) here of particular interest. + +Also, +Haskell programs consume native libraries directly via an FFI mechanism. +That is, there is no "Haskell extension module" to build. + +## Obelisk + +Obelisk apparently works for pure-Haskell applications. +Obelisk itself does not attempt to support build native libraries. +Instead, it relies on the native capabilities of Cabal to satisfy these dependencies. +Cabal relies on the standard ``pkg-config`` tool to find headers and shared objects for native libraries. + +We learned a lot about cross-compiling native library dependencies while trying to make CPython work. +Thus we already have many of the native library dependencies cross-compiling in a maintainable way. + +What remains here is to integrate the Haskell build with these libraries. +This is where our efforts are currently directed. + +Additionally, +Obelisk is based on a system called "Reflex-FRP" which brings a DOM-based Functional Reactive Programming UI layer. +It is likely that if Obelisk is our chosen solution then we will build the UI layer for the application using this and presenting the result in a web view. diff --git a/crossenv/flake.nix b/crossenv/flake.nix index fdaa04f614d661dce5f782f9f72b2931d62785d1..c9dd455f9719237f0f9d75f9208c6af40ee699c8 100644 --- a/crossenv/flake.nix +++ b/crossenv/flake.nix @@ -81,6 +81,9 @@ # # build-pip install a-build-dependency # pip wheel somepackagetobuild + + # For Tahoe-LAFS dependencies you really want to have cffi: + build-pip install cffi ''; shellHook = '' diff --git a/zfec.nix b/zfec.nix new file mode 100644 index 0000000000000000000000000000000000000000..fe59458d8d8cc4eee4f1c6bd0f89f6c926dc87ae --- /dev/null +++ b/zfec.nix @@ -0,0 +1,35 @@ +{ lib +, buildPythonPackage +, fetchPypi +}: + +buildPythonPackage rec { + pname = "zfec"; + version = "1.5.7.2"; + + src = fetchPypi { + inherit pname version; + sha256 = "sha256-TuUZvg3MfaLohIK8/Av5d6Ql4dfoJ4z1u7uNAPiir7Y="; + }; + + checkInputs = [ ]; + checkPhase = "python -m unittest zfec.test.test_zfec"; + + pythonImportsCheck = [ "zfec" ]; + + meta = with lib; { + homepage = "https://github.com/tahoe-lafs/zfec"; + description = "Zfec, a fast erasure codec which can be used with the command-line, C, Python, or Haskell"; + longDescription = '' + Fast, portable, programmable erasure coding a.k.a. "forward + error correction": the generation of redundant blocks of + information such that if some blocks are lost then the + original data can be recovered from the remaining blocks. The + zfec package includes command-line tools, C API, Python API, + and Haskell API. + ''; + license = licenses.gpl2Plus; + maintainers = with maintainers; [ prusnak ]; + }; + +}