Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
# Summary
We have explored a couple of tools for Python Android development.
Overall the tools appear to fall far short of what is necessary to achieve our goals in a timely manner.
Additionally, the solutions that must be created to use these tools are ultimately of low and limited-term value.
We have also explored a couple approaches for Haskell Android development.
These are also not mainstream, highly-polished tools.
However, what they accomplish they do so with less complexity than the equivalent Python tools.
Building on these tools also creates solutions that are potentially of high and long-term value.
# The Challenge
To build a Tahoe-LAFS-based mobile application there are N challenges to overcome:
1. Create a user interface that allows users to interact with the system
2. Interact with Tahoe-LAFS using its cryptographic and network protocols
3. Build and distribute the entire client system
For most of our investigation we took as an assumption that (1) would be solved by using standard mobile system UI development tools.
For example, native Android GUI functionality is readily available to applications written in Java or Kotlin.
Thus, we focused on (2) and (3).
# Python Tools
The motivation behind investigating Python is that it almost entirely solves challenge (2).
There is an existing Python-based Tahoe-LAFS implementation including all of the pieces we need.
Given this, we investigated how to use Python and overcome challenge (3).
Challenge (3) can be further broken down into at least two parts:
1. Build native library dependencies
2. Build CPython extension modules exposing those native library dependencies to Python
We looked at five approaches for this:
1. Kivy
2. Chaquopy
3. crossenv
4. nix
5. direct builds
## Kivy
Kivy apparently works for pure-Python applications.
However it does not attempts to support building Python extension modules for Android.
Tahoe-LAFS has several such dependencies, including:
* zfec
* python-cryptography
* python-challenge-bypass-ristretto
Therefore,
though Kivy is the most popular and well-known of these tools,
it does not actually supply any solution to challenge (3).
Additionally,
Kivy uses a fully custom build tool-chain for creating Android packages (APKs).
This adds substantially to the complexity of the system
This translated into spending much longer trying to learn how Kivy builds work and whether it is possible to customize them as necessary for our purposes.
## Chaquopy
Our experience with Chaquopy was similar.
Though it does not use a custom build system,
it is still not capable of building the Python extension modules we require.
The Chaquopy creator/maintainer has built several Python extension modules and makes these builds available.
However,
because Chaquopy itself does not include the tools for performing these builds,
several problems remain:
* The modules supply by the creator/maintainer are increasingly out of date.
* Some dependencies of Tahoe-LAFS are also missing from this set.
* Finally,
this path implies using non-reproducible binary artifacts from a relatively unknown source.
We feel this is an untenable premise for a security- and privacy-focused application.
## crossenv
crossenv proposes *only* to solve the cross-compilation problem.
This means it would be a complement to Kivy or Chaquopy.
Our efforts with crossenv quickly yielded success for *most* dependencies of Tahoe-LAFS.
The following modules cross-compiled (though we did not test the results):
* python-challenge-bypass-ristretto
Note that you at least need to install cffi in the
build-python environment. Maybe some other deps require
this as well..
* bcrypt
* cbor2
* netifaces
* pycddl
* zope.interface
* pyyaml
However, the following modules failed to cross-compile:
* pynacl
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
* cryptography
ld: error: unable to find library -lpthread
These are substantial dependencies and it is not feasible to use Tahoe-LAFS without them.
The cross-compilation failures come from somewhere within the CPython system for building extension modules.
We spent significant time trying to understand this system
(for crossenv but also in general as part of this spike effort).
Our conclusion is that CPython is ill-suited to supporting cross-compilation and fixing these build failures might amount to doing significant implementation work on CPython itself.
Given our other attempts to integrate CPython development work upstream,
we have limited hope of successfully integrating fixes to the cross-compilation system.
## nix / nixpkgs
nixpkgs itself has robust support for cross-compilation as a core feature.
It appears as though someone probably used this to support our use-case at some point in time.
However, it also appears as though this feature has "bitrotted".
We were able to build many C library dependencies for Android.
However, all of the Python extension modules fail to build due to issues with Python's cross-compilation support.
## direct builds
This approach involved carefully hand-crafting cross-compilation instructions for individual build dependencies.
Because of the cost in effort of this approach we only tried to apply it to one of the simplest extension modules we require: zfec.
We were eventually about to build the zfec C and Python modules for Android.
We did not test any of the build outputs.
The build setup is very fragile and requires a number of careful fix-ups at the end.
It also took several days to complete.
Most of this setup is unique to zfec and cannot be generalized to the other dependencies we need to build.
Additionall,y other dependencies themselves have more dependencies.
For example, python-cryptography depends on the entire Rust toolchain.
We anticipate applying this approach to other dependencies would be prohibitive in the amount of time required.
# Haskell Tools
The motivation behind investigating Haskell is that it represents a technology choice that will be valuable in the long term.
The aspects which this conclusion are based on are:
a. Its expressive type system makes developing complex software less complicated.
b. Its runtime CPU efficiency is comparable to that of JVM-based software
(the typical runtime for Android applications).
c. We are aware that some people are using Haskell for Android apps in the real world.
Considering the difficulty we found in using Python to address challenge (3) we find item (c) here of particular interest.
Also,
Haskell programs consume native libraries directly via an FFI mechanism.
That is, there is no "Haskell extension module" to build.
## Obelisk
Obelisk apparently works for pure-Haskell applications.
Obelisk itself does not attempt to support build native libraries.
Instead, it relies on the native capabilities of Cabal to satisfy these dependencies.
Cabal relies on the standard ``pkg-config`` tool to find headers and shared objects for native libraries.
We learned a lot about cross-compiling native library dependencies while trying to make CPython work.
Thus we already have many of the native library dependencies cross-compiling in a maintainable way.
What remains here is to integrate the Haskell build with these libraries.
This is where our efforts are currently directed.
Additionally,
Obelisk is based on a system called "Reflex-FRP" which brings a DOM-based Functional Reactive Programming UI layer.
It is likely that if Obelisk is our chosen solution then we will build the UI layer for the application using this and presenting the result in a web view.