Changes

tex/chapters/8_experiments.tex

@@ -73,14 +73,15 @@ We used a \SI{10}{cm} coaxial cable to connect the antennas to the cards, which
 %    \item Bildergrid und Video
 %\end{itemize}
 
-The first experiment evaluates the indoor precision of FTM distance measurements given different hardware configurations.
-While \etal{Ibrahim} \cite{ibrahim2018verification} already verified the precision of the Intel~AC~8260 card in great detail, our setup differs from theirs and requires anew evaluation.
+The first experiment evaluates the indoor precision and accuracy of FTM distance measurements given different hardware configurations.
+While \etal{Ibrahim} \cite{ibrahim2018verification} already verified the precision of the \intelOld card in great detail, our setup differs from theirs and requires anew evaluation.
 In contrast to \etal{Ibrahim} we use smartphones as receivers and two different cards with different firmware versions as senders.
 Additionally, it is unclear how the external antennas affect the measurements.
 For these reasons we did a static distance measurement experimental setup to confirm that the combination of Pixel devices and Intel cards provide reliable values. 
 
+% TODO fig mit shematischen aufbau
 
-Our test setup consist of 10 measurement points evenly spaced on a straight line with a distance of \SI{2}{m}.
+Our test setup consist of $10$ measurement points evenly spaced at a distance of \SI{2}{m} on a straight line.
 The closest point to the AP is \SI{2}{m} away and the furthest \SI{20}{m}.
 At every point each phone is placed on a stand.
 Around 140 FTM measurements are recorded, which corresponds to a measure period of \SI{30}{s} per point.
@@ -91,8 +92,8 @@ While recording measurements for \SI{30}{s} at a single point is not realistic i
 
 The whole experiment was deployed in the hallway of our university.
 Each distance measurement is performed with every hardware combination.
-On the receiving side we used the Google Pixel 2 XL and Pixel 3a.
-On the sending side we used the Intel AC 8260 and 9462 with internal and external \SI{2}{dBi} antennas.
+On the receiving side we used the Google \pixelOld and \pixelNew.
+On the sending side we used the \intelBoth with internal and external \SI{2}{dBi} omnidirectional antennas.
 The antenna geometry and properties of the internal antenna are unknown.
 In total there are eight phone AP combinations.
 
@@ -106,28 +107,70 @@ In total there are eight phone AP combinations.
 \autoref{fig:DistMeasMeanNucPixel}~\subref{fig:DistMeasMeanNucPixel:a} shows the average measured distance per smartphone in respect to the ground truth distance.
 Likewise, \autoref{fig:DistMeasMeanNucPixel}~\subref{fig:DistMeasMeanNucPixel:b} depicts the average measured distance per access point.
 The corresponding values of these figures are shown in \autoref{tab:distvaluesPixels} and \autoref{tab:distvaluesNUCs}.
+We compute the mean over the 140 FTM measurements denoted as $\bar{d}$ and its standard deviation.
+Because $\bar{d}$ can be larger or smaller than the true distance we use the difference between $\bar{d}$ and the true distance as error metric.
+However, when its necessary to quantify the error regardless of its direction the absolute difference is used.
 Interestingly, both the wireless cards and the Pixel devices exhibit some similar tendency regarding the measurement error.
 
-As seen in \autoref{fig:DistMeasMeanNucPixel}~\subref{fig:DistMeasMeanNucPixel:a} the Pixel 3a tends to underestimate the distance.
+As seen in \autoref{fig:DistMeasMeanNucPixel}~\subref{fig:DistMeasMeanNucPixel:a} the \pixelOld tends to underestimate the distance.
 Only at \SI{6}{m} and \SI{10}{m} the estimated distance is slightly larger compared to the true distance.
 The overall error is mostly negative and the mean absolute error is $\SI{1}{m}$.
-Contrarily, the Pixel 2 XL tends to overestimates the distance compared to the groundtruth distance.
+Contrarily, the \pixelNew tends to overestimate the distance compared to the groundtruth distance.
 Here the mean absolute error is $\SI{1.4}{m}$.
-However, at the \SI{16}{m} mark the measured mean distance of the Pixel 2 XL significantly increases.
-Computing the mean absolute error only in the interval of $[\SI{2}{m}, \SI{16}{m}]$ reduces the Pixel 2 XL error to $\SI{0.6986}{m}$, while the Pixel 3a error changes negligible.
+However, at the \SI{16}{m} mark the measured mean distance of the \pixelNew significantly increases.
+Computing the mean absolute error only in the interval of $[\SI{2}{m}, \SI{16}{m}]$ reduces the \pixelNew error to $\SI{0.6986}{m}$, at the same time the \pixelOld error changes negligible.
+While on average the standard deviation of the distance measurements are quite similar for both devices, the standard deviation for the \pixelNew is more stable.
 
-At \SI{16}{m} the Pixel 3a stops to underestimate the distance and the measurements at \SI{18}{m} and \SI{20}{m} are quite close to the true distance.
-In contrast, the Pixel 2 XL starts to increasingly overestimate the true distance which results in large error values ($\approx \SI{4}{m}$).
-The same behavior is observable for the Intel AC 8260 and 9460 cards.
+At \SI{16}{m} the \pixelOld stops to underestimate the distance and the measurements at \SI{18}{m} and \SI{20}{m} are quite close to the true distance.
+In contrast, the \pixelNew starts to increasingly overestimate the true distance which results in large error values ($\approx \SI{4}{m}$).
+The same behavior is observable for the \intelBoth cards.
 Again at \SI{16}{m} both cards start to overestimate the true distance.
 
 For distances smaller than \SI{16}{m} the \intelOld also underestimates the distance with a mean absolute error of \SI{1.11}{m} in that range.
-Like the \pixelOld the error increases for larger distances, however, somewhat smaller with $\approx \SI{2}{m}$.
+Like the \pixelNew the error increases for larger distances, however, somewhat smaller with $\approx \SI{2}{m}$.
 In total the \intelNew card tends to provide an accurate distance estimate but has some outliers at \SI{6}{m} and \SI{10}{m} but never underestimates the true distance.
 
 While the mean distance over many measurements is relevant for stationary measure points, in our scenario a pedestrian is moving with the smartphone.
 Therefore, only one or a few measurements can be observed at a given position.
-A more expressive visualization for this scenario is given with the CDF graph in \autoref{fig:DistMeasMeanNucPixel}~\subref{fig:DistMeasMeanNucPixel:c}.
+A more expressive visualization for this scenario is given with the CDF graph in \autoref{fig:DistMeasMeanNucPixel}~\subref{fig:DistMeasMeanNucPixel:c} which allows to reason about the underlying error distribution.
+
+Most striking is the curve of the \intelOld and \pixelOld combination with internal antenna (red dashed line).
+Firstly, about 80\% of the measurements have a negative error, \ie underestimate the true distance.
+Secondly, the curve indicates that the error distribution is a Gaussian mixture distribution with two modes at \SI{-3.201}{m} and \SI{0.0879}{m}, whereas the mode at \SI{-3.201}{m} provides about 60\% of the probability mass.
+The multimodality is greatly reduced by using the external antenna (orange dashed line), but still about 70\% of the measurements are smaller than the true distance.
+This indicates that the performance of this particular device combination could be improved by adding a constant factor of around \SI{1}{m}.
+
+However, using the same card together with the \pixelNew (red line) the error is already much smaller and only a small portion is negative.
+In this case no constant offset would significantly improve the measurements.
+Surprisingly, the error distribution of the \intelOld and \pixelNew combination with added external antennas (orange line) dramatically changes compared to the internal antenna.
+There are more negative error values and large positive errors are introduced, which where non-existent with the internal antenna setup.
+
+As a result no clear recommendation to either use the internal or external antennas with the \intelOld can be stated.
+While the external antenna significantly reduces negative errors of the \pixelOld it introduces large positive errors for the \pixelNew.
+At the same time the \pixelOld would perform reasonable well without external antennas.
+Underestimated distances are somewhat surprising, as only overestimated distances are expected with time based methods.
+Furthermore, underestimated measurements could result in negative distances which are hard to reason about.
+Therefore, one possible argument to choose the external antennas is to reduce the negative error while accepting some more positive errors.
+
+
+In the case of the \intelNew card the effect of the antennas is marginal.
+The error of the \pixelOld (blue dashed line) is primarily positive and mostly less than \SI{3}{m}.
+Using external antennas actually worsened the measurements producing much more negative errors.
+Note that this behavior is on the contrary to the error distributions of the \intelOld card.
+
+In contrast, the error of the \pixelNew with the \intelNew card is changes only insignificant with the antennas.
+While 50\% of the error is smaller than \SI{1.5}{m} with the external antenna (light green line), the error is smaller than \SI{2.5}{m} in the same range for the internal antenna (blue line).
+
+%TODO RSSI erwähnen
+
+In sum, with these results no clear tendency could be observed whether to use the internal or external antennas to reduce the error of the FTM measurements.
+Of course this is only true for this specific experiment, for different environmental conditions or sender and receiver positions the results may vary.
+More significant is the choice of the particular wireless card.
+Here, the \intelNew mainly gives better results compared to the \intelOld, \ie most of the errors are positive and in a range of up to \SI{5}{m}.
+Both the \pixelBoth produce similar results, but the \pixelOld tends to underestimate the distance more compared to the \pixelNew.
+
+Opposed to \etal{Ibrahim} \cite{ibrahim2018verification} findings no single constant offset which significantly improves the measurements across all device combinations could be found in our tests.
+However, the overall error of the device combinations is reasonable small and its distribution is mostly Gaussian-like, which justifies the basic applicability of the technique and devices for indoor positioning.
 
 \begin{figure}[ht]
 
@@ -152,18 +195,18 @@ A more expressive visualization for this scenario is given with the CDF graph in
 \renewcommand{\arraystretch}{1.2}
 \begin{tabular}{@{}RRRRcRRR@{}}
 \toprule
-\theadbf{GT dist in m} & \multicolumn{3}{c}{\bfseries Pixel 2 XL}  & \phantom{a} & \multicolumn{3}{c}{\bfseries Pixel 3a}  \\ \cmidrule{2-4} \cmidrule{6-8}
+\theadbf{GT dist in m} & \multicolumn{3}{c}{\bfseries \pixelOld}  & \phantom{a} & \multicolumn{3}{c}{\bfseries \pixelNew}  \\ \cmidrule{2-4} \cmidrule{6-8}
 ~            & \thead{$\bar{d}$} & \thead{$\sigma$} & \thead{error} && \thead{$\bar{d}$} & \thead{$\sigma$} & \thead{error} \\ \midrule
-2  & 2.472  & 1.165 &  0.472 && 0.032  & 1.615 & -1.968 \\ 
-4  & 3.976  & 1.755 & -0.024 && 2.645  & 1.834 & -1.355 \\ 
-6  & 6.605  & 1.978 &  0.605 && 6.577  & 1.780 &  0.577 \\ 
-8  & 8.780  & 1.510 &  0.780 && 6.861  & 1.506 & -1.139 \\ 
-10 & 11.520 & 1.522 &  1.520 && 10.454 & 2.176 &  0.454 \\ 
-12 & 12.096 & 1.582 &  0.096 && 10.342 & 1.875 & -1.658 \\ 
-14 & 14.327 & 1.870 &  0.327 && 12.866 & 2.309 & -1.134 \\ 
-16 & 17.765 & 1.488 &  1.765 && 15.992 & 0.879 & -0.008 \\ 
-18 & 22.569 & 1.458 &  4.569 && 18.962 & 1.491 &  0.962 \\ 
-20 & 24.058 & 1.820 &  4.058 && 20.742 & 1.318 &  0.742 \\
+ 2 &  0.032 & 1.615 & -1.968 &&  2.472 & 1.165 &  0.472 \\ 
+ 4 &  2.645 & 1.834 & -1.355 &&  3.976 & 1.755 & -0.024 \\ 
+ 6 &  6.577 & 1.780 &  0.577 &&  6.605 & 1.978 &  0.605 \\ 
+ 8 &  6.861 & 1.506 & -1.139 &&  8.780 & 1.510 &  0.780 \\ 
+10 & 10.454 & 2.176 &  0.454 && 11.520 & 1.522 &  1.520 \\ 
+12 & 10.342 & 1.875 & -1.658 && 12.096 & 1.582 &  0.096 \\ 
+14 & 12.866 & 2.309 & -1.134 && 14.327 & 1.870 &  0.327 \\ 
+16 & 15.992 & 0.879 & -0.008 && 17.765 & 1.488 &  1.765 \\ 
+18 & 18.962 & 1.491 &  0.962 && 22.569 & 1.458 &  4.569 \\ 
+20 & 20.742 & 1.318 &  0.742 && 24.058 & 1.820 &  4.058 \\ 
 \bottomrule
 \end{tabular}
 \caption{TODO}
@@ -176,7 +219,7 @@ A more expressive visualization for this scenario is given with the CDF graph in
 \renewcommand{\arraystretch}{1.2}
 \begin{tabular}{@{}RRRRcRRR@{}}
 \toprule
-\theadbf{GT dist in m} & \multicolumn{3}{c}{\bfseries Intel AC 8260}  & \phantom{a} & \multicolumn{3}{c}{\bfseries Intel AC 9460}  \\ \cmidrule{2-4} \cmidrule{6-8}
+\theadbf{GT dist in m} & \multicolumn{3}{c}{\bfseries \intelOld}  & \phantom{a} & \multicolumn{3}{c}{\bfseries \intelNew}  \\ \cmidrule{2-4} \cmidrule{6-8}
 ~            & \thead{$\bar{d}$} & \thead{$\sigma$} & \thead{error} && \thead{$\bar{d}$} & \thead{$\sigma$} & \thead{error} \\ \midrule
 2  &  0.221 & 1.460 & -1.779 &&  2.282 & 1.724 & 0.282 \\ 
 4  &  2.175 & 0.541 & -1.825 &&  4.446 & 1.973 & 0.446 \\ 

tex/misc/keywords.tex

@@ -20,9 +20,11 @@
 
 \newcommand{\pixelOld}{Pixel~2~XL\xspace}
 \newcommand{\pixelNew}{Pixel~3a\xspace}
+\newcommand{\pixelBoth}{Pixel~2~XL and~3a\xspace}
 
 \newcommand{\intelOld}{Intel~AC~8260\xspace}
-\newcommand{\intelNew}{Intel~AC~9460\xspace}
+\newcommand{\intelNew}{Intel~AC~9462\xspace}
+\newcommand{\intelBoth}{Intel~AC~8260 and~9462\xspace}
 
 % keyword macros
 \newcommand{\docIBeacon}{iBeacon}