From abe7bdd6d2682f0ef79f82bffe2ea83f20759511 Mon Sep 17 00:00:00 2001
From: Jean-Paul Calderone <exarkun@twistedmatrix.com>
Date: Mon, 12 Dec 2022 10:00:16 -0500
Subject: [PATCH] some more pieces and an attempt at a summary
---
Notes.rst | 33 ++++++---
Summary.md | 170 +++++++++++++++++++++++++++++++++++++++++++++
crossenv/flake.nix | 3 +
zfec.nix | 35 ++++++++++
4 files changed, 233 insertions(+), 8 deletions(-)
create mode 100644 Summary.md
create mode 100644 zfec.nix
diff --git a/Notes.rst b/Notes.rst
index 5189eab..6129cd1 100644
--- a/Notes.rst
+++ b/Notes.rst
@@ -1,3 +1,6 @@
+notes
+=====
+
* Build a PrivateStorage-related Android application
* Build an app that can interact with magic-folders somehow
@@ -208,20 +211,34 @@
* python-challenge-bypass-ristretto
- * crossenv can do it!
+ * crossenv can do it! Mostly!
See crossenv subdirectory.
- * However, crossenv can't do python-challenge-bypass-ristretto
- or python-cryptography.
+ These work:
+
+ * python-challenge-bypass-ristretto
+
+ Note that you at least need to install cffi in the
+ build-python environment. Maybe some other deps require
+ this as well..
+
+ * bcrypt
+ * cbor2
+ * netifaces
+ * pycddl
+ * zope.interface
+ * pyyaml
+
+ These fail:
- These seem to have cffi as a build-time dependency.
- Cross-env gives the build pip env the host cffi .so so it fails.
+ * pynacl
- Ah, but you can clobber this with::
+ configure: error: cannot run C compiled programs.
+ If you meant to cross compile, use `--host'.
- build-pip install cffi
+ * cryptography
- And somehow this gets used instead.
+ ld: error: unable to find library -lpthread
* Integrate a Python Tahoe-LAFS runtime with a GUI
diff --git a/Summary.md b/Summary.md
new file mode 100644
index 0000000..6d13ef0
--- /dev/null
+++ b/Summary.md
@@ -0,0 +1,170 @@
+# Summary
+
+We have explored a couple of tools for Python Android development.
+Overall the tools appear to fall far short of what is necessary to achieve our goals in a timely manner.
+Additionally, the solutions that must be created to use these tools are ultimately of low and limited-term value.
+
+We have also explored a couple approaches for Haskell Android development.
+These are also not mainstream, highly-polished tools.
+However, what they accomplish they do so with less complexity than the equivalent Python tools.
+Building on these tools also creates solutions that are potentially of high and long-term value.
+
+# The Challenge
+
+To build a Tahoe-LAFS-based mobile application there are N challenges to overcome:
+
+1. Create a user interface that allows users to interact with the system
+2. Interact with Tahoe-LAFS using its cryptographic and network protocols
+3. Build and distribute the entire client system
+
+For most of our investigation we took as an assumption that (1) would be solved by using standard mobile system UI development tools.
+For example, native Android GUI functionality is readily available to applications written in Java or Kotlin.
+Thus, we focused on (2) and (3).
+
+# Python Tools
+
+The motivation behind investigating Python is that it almost entirely solves challenge (2).
+There is an existing Python-based Tahoe-LAFS implementation including all of the pieces we need.
+Given this, we investigated how to use Python and overcome challenge (3).
+
+Challenge (3) can be further broken down into at least two parts:
+
+1. Build native library dependencies
+2. Build CPython extension modules exposing those native library dependencies to Python
+
+We looked at five approaches for this:
+
+1. Kivy
+2. Chaquopy
+3. crossenv
+4. nix
+5. direct builds
+
+## Kivy
+
+Kivy apparently works for pure-Python applications.
+However it does not attempts to support building Python extension modules for Android.
+Tahoe-LAFS has several such dependencies, including:
+
+* zfec
+* python-cryptography
+* python-challenge-bypass-ristretto
+
+Therefore,
+though Kivy is the most popular and well-known of these tools,
+it does not actually supply any solution to challenge (3).
+Additionally,
+Kivy uses a fully custom build tool-chain for creating Android packages (APKs).
+This adds substantially to the complexity of the system
+This translated into spending much longer trying to learn how Kivy builds work and whether it is possible to customize them as necessary for our purposes.
+
+## Chaquopy
+
+Our experience with Chaquopy was similar.
+Though it does not use a custom build system,
+it is still not capable of building the Python extension modules we require.
+The Chaquopy creator/maintainer has built several Python extension modules and makes these builds available.
+However,
+because Chaquopy itself does not include the tools for performing these builds,
+several problems remain:
+
+* The modules supply by the creator/maintainer are increasingly out of date.
+* Some dependencies of Tahoe-LAFS are also missing from this set.
+* Finally,
+ this path implies using non-reproducible binary artifacts from a relatively unknown source.
+ We feel this is an untenable premise for a security- and privacy-focused application.
+
+## crossenv
+
+crossenv proposes *only* to solve the cross-compilation problem.
+This means it would be a complement to Kivy or Chaquopy.
+
+Our efforts with crossenv quickly yielded success for *most* dependencies of Tahoe-LAFS.
+The following modules cross-compiled (though we did not test the results):
+
+* python-challenge-bypass-ristretto
+
+ Note that you at least need to install cffi in the
+ build-python environment. Maybe some other deps require
+ this as well..
+
+* bcrypt
+* cbor2
+* netifaces
+* pycddl
+* zope.interface
+* pyyaml
+
+However, the following modules failed to cross-compile:
+
+* pynacl
+
+ configure: error: cannot run C compiled programs.
+ If you meant to cross compile, use `--host'.
+
+* cryptography
+
+ ld: error: unable to find library -lpthread
+
+These are substantial dependencies and it is not feasible to use Tahoe-LAFS without them.
+The cross-compilation failures come from somewhere within the CPython system for building extension modules.
+We spent significant time trying to understand this system
+(for crossenv but also in general as part of this spike effort).
+Our conclusion is that CPython is ill-suited to supporting cross-compilation and fixing these build failures might amount to doing significant implementation work on CPython itself.
+Given our other attempts to integrate CPython development work upstream,
+we have limited hope of successfully integrating fixes to the cross-compilation system.
+
+## nix / nixpkgs
+
+nixpkgs itself has robust support for cross-compilation as a core feature.
+It appears as though someone probably used this to support our use-case at some point in time.
+However, it also appears as though this feature has "bitrotted".
+We were able to build many C library dependencies for Android.
+However, all of the Python extension modules fail to build due to issues with Python's cross-compilation support.
+
+## direct builds
+
+This approach involved carefully hand-crafting cross-compilation instructions for individual build dependencies.
+Because of the cost in effort of this approach we only tried to apply it to one of the simplest extension modules we require: zfec.
+
+We were eventually about to build the zfec C and Python modules for Android.
+We did not test any of the build outputs.
+The build setup is very fragile and requires a number of careful fix-ups at the end.
+It also took several days to complete.
+Most of this setup is unique to zfec and cannot be generalized to the other dependencies we need to build.
+Additionall,y other dependencies themselves have more dependencies.
+For example, python-cryptography depends on the entire Rust toolchain.
+We anticipate applying this approach to other dependencies would be prohibitive in the amount of time required.
+
+# Haskell Tools
+
+The motivation behind investigating Haskell is that it represents a technology choice that will be valuable in the long term.
+The aspects which this conclusion are based on are:
+
+a. Its expressive type system makes developing complex software less complicated.
+b. Its runtime CPU efficiency is comparable to that of JVM-based software
+ (the typical runtime for Android applications).
+c. We are aware that some people are using Haskell for Android apps in the real world.
+
+Considering the difficulty we found in using Python to address challenge (3) we find item (c) here of particular interest.
+
+Also,
+Haskell programs consume native libraries directly via an FFI mechanism.
+That is, there is no "Haskell extension module" to build.
+
+## Obelisk
+
+Obelisk apparently works for pure-Haskell applications.
+Obelisk itself does not attempt to support build native libraries.
+Instead, it relies on the native capabilities of Cabal to satisfy these dependencies.
+Cabal relies on the standard ``pkg-config`` tool to find headers and shared objects for native libraries.
+
+We learned a lot about cross-compiling native library dependencies while trying to make CPython work.
+Thus we already have many of the native library dependencies cross-compiling in a maintainable way.
+
+What remains here is to integrate the Haskell build with these libraries.
+This is where our efforts are currently directed.
+
+Additionally,
+Obelisk is based on a system called "Reflex-FRP" which brings a DOM-based Functional Reactive Programming UI layer.
+It is likely that if Obelisk is our chosen solution then we will build the UI layer for the application using this and presenting the result in a web view.
diff --git a/crossenv/flake.nix b/crossenv/flake.nix
index fdaa04f..c9dd455 100644
--- a/crossenv/flake.nix
+++ b/crossenv/flake.nix
@@ -81,6 +81,9 @@
#
# build-pip install a-build-dependency
# pip wheel somepackagetobuild
+
+ # For Tahoe-LAFS dependencies you really want to have cffi:
+ build-pip install cffi
'';
shellHook = ''
diff --git a/zfec.nix b/zfec.nix
new file mode 100644
index 0000000..fe59458
--- /dev/null
+++ b/zfec.nix
@@ -0,0 +1,35 @@
+{ lib
+, buildPythonPackage
+, fetchPypi
+}:
+
+buildPythonPackage rec {
+ pname = "zfec";
+ version = "1.5.7.2";
+
+ src = fetchPypi {
+ inherit pname version;
+ sha256 = "sha256-TuUZvg3MfaLohIK8/Av5d6Ql4dfoJ4z1u7uNAPiir7Y=";
+ };
+
+ checkInputs = [ ];
+ checkPhase = "python -m unittest zfec.test.test_zfec";
+
+ pythonImportsCheck = [ "zfec" ];
+
+ meta = with lib; {
+ homepage = "https://github.com/tahoe-lafs/zfec";
+ description = "Zfec, a fast erasure codec which can be used with the command-line, C, Python, or Haskell";
+ longDescription = ''
+ Fast, portable, programmable erasure coding a.k.a. "forward
+ error correction": the generation of redundant blocks of
+ information such that if some blocks are lost then the
+ original data can be recovered from the remaining blocks. The
+ zfec package includes command-line tools, C API, Python API,
+ and Haskell API.
+ '';
+ license = licenses.gpl2Plus;
+ maintainers = with maintainers; [ prusnak ];
+ };
+
+}
--
GitLab