168 lines
14 KiB
TeX
168 lines
14 KiB
TeX
\section{Experiments}
|
|
|
|
% all paths we evaluated
|
|
\begin{figure}
|
|
\input{gfx/eval/paths}
|
|
\caption{The four paths that were part of the evaluation.
|
|
Starting positions are marked with black circles.
|
|
For a better visualisation they were slightly shifted to avoid overlapping.}
|
|
%\commentByFrank{font war korrekt, aber die groesse war zu gross im vgl. zu den anderen}
|
|
\label{fig:paths}
|
|
\end{figure}
|
|
%
|
|
The experiments were carried out on all four floors of the faculty building.
|
|
Each floor is about \SI{77}{\meter} x \SI{55}{\meter} in size, with a ceiling height of about \SI{3.0}{\meter}.
|
|
To resemble real-world conditions, the evaluation took place during an in-house exhibition.
|
|
Thus, many places were crowded and Wi-Fi signals attenuated.
|
|
As can be seen in fig. \ref{fig:paths}, we arranged 4 distinct walks, covering different distances, critical sections and uncertain decisions leading to multimodalities.
|
|
The ground truth is measured by recording a timestamp at marked spots on the walking route.
|
|
When passing a marker, the pedestrian clicked a button on the smartphone application.
|
|
Between two consecutive points, a constant movement speed is assumed.
|
|
Thus, the ground truth might not be \SI{100}{\percent} accurate, but fair enough for error measurements.
|
|
The approximation error is then calculated by comparing the interpolated ground truth position with the current estimation.
|
|
Especially in the context of smoothing, it is also very interesting to exclude the temporal delay and measure only the positional difference between estimated and ground truth path.
|
|
This provides a statement about the extent to which the smoothed path superficially improves compared to the filtered one.
|
|
|
|
All walks start with a uniform distribution (random position and heading) as prior for $\mStateVec_0$.
|
|
To allow the system to stabilise its initial state, the first few estimations are omitted from error calculations.
|
|
Even though the error during the following few seconds is expected to be much higher than the error when starting with a well known initial position and heading.
|
|
|
|
The measurements were recorded using a Motorola Nexus 6 and a Samsung Galaxy S5.
|
|
As the Galaxy's \docWIFI{} cannot be limited to the \SI{2.4}{\giga\hertz} band only, its scans take much longer than those of the Nexus: \SI{3500}{\milli\second} vs. \SI{600}{\milli\second}.
|
|
Additionally, the Galaxy's barometer sensor provides far more inaccurate and less frequent readings than the Nexus does.
|
|
This results in a better localisation using the Nexus smartphone.
|
|
The computation for both filtering and smoothing was done offline using the aforementioned \mbox{CONDENSATION} algorithm and multinomal (cumulative) resampling.
|
|
For each path we deployed 10 MC runs using \SI{2500}{} particles. BS uses $500$ sample realisations drawn with a cumulative frequency.
|
|
%However, the filter itself would be fast enough to run on the smartphone itself ($ \approx \SI{100}{\milli\second} $ per transition, single-core Intel\textsuperscript{\textregistered} Atom{\texttrademark} C2750).
|
|
%The computational times of the different smoothing algorithm will be discussed later.
|
|
Unless explicitly stated, the state was estimated using the weighted arithmetic mean of the particle set.
|
|
|
|
As mentioned earlier, the position of all \docAP{}s (about 5 per floor) is known beforehand.
|
|
Due to legal terms, we are not allowed to depict their positions and therefore omit this information within the figures.
|
|
Additionally, we used five \docIBeacon{}s for slight enhancements in some areas.
|
|
The empirically chosen values for \docWIFI{} were $P_{0_{\text{wifi}}} = \SI{-46}{\dBm}, \mPLE_{\text{wifi}} = \SI{2.7}{}, \mWAF_{\text{wifi}} = \SI{8}{\dB}$, and $\mPLE_{\text{ib}} = \SI{1.5}{}$ for the \docIBeacon{}s, respectively.
|
|
The system was tested by omitting any time-consuming calibration processes for those values.
|
|
We therefore expect the localisation process to perform generally worse compared to standard fingerprinting methods \cite{Ville09}.
|
|
However, smoothing will often compensate for those poorly chosen system parameters.
|
|
|
|
For the filtering we used $\sigma_\text{wifi} = \sigma_\text{ib} = 8.0$ as uncertainties, both growing with each measurement's age.
|
|
While the pressure change was assumed to be \SI{0.105}{$\frac{\text{\hpa}}{\text{\meter}}$}, all other barometer-parameters are determined automatically (see \ref{sec:eval}).
|
|
The step size $\mStepSize$ for the transition was configured to be \SI{70}{\centimeter} with an allowed derivation of \SI{10}{\percent}. The heading deviation was set to \SI{25}{\degree}.
|
|
Edges departing from the pedestrian's destination are downvoted using $\mUsePath = 0.9$.
|
|
For smoothing we set $\sigma^2_{\text{turn}} = \SI{5}{\degree}$ and $\sigma^2_z = \SI{0.25}{\centimeter}$.
|
|
If walking or unknown are the current activities, $ \mu_{\text{step}} = \SI{0.7}{\meter}$, $ \mu_{\text{step}} = \SI{0.5}{}$ and $ \mu_z = \SI{0.0}{\meter}$ are used.
|
|
Walking upstairs sets $ \mu_{\text{step}} = \SI{0.4}{\meter}$, $ \sigma_{\text{step}}^2 = \SI{0.2}{}$ and $ \mu_z = \SI{-0.3}{\meter}$, otherwise $ \mu_{\text{step}} = \SI{0.5}{\meter}$, $ \sigma_{\text{step}}^2 = \SI{0.3}{}$ and $ \mu_z = \SI{0.3}{\meter}$ for walking downstairs.
|
|
|
|
|
|
%kurz zeigen das activity recognition was bringt
|
|
%\begin{figure}
|
|
% \input{gfx/activity/activity_over_time}
|
|
% \caption{The activities recognized for path 4. The misdetection in seg. 2 is cause by faulty pressure readings.}
|
|
% \label{fig:activityRecognition}
|
|
%\end{figure}
|
|
%By adding the activity recognition the approximation error of the filter decreases by an average of \SI{XX}{\centimeter} for all 4 paths.
|
|
%Due to this additional knowledge, the state transition samples mostly %depending upon the current activity and therefore limits the possibility of false floor changes.
|
|
%Fig. \ref{fig:activityRecognition} shows the recognized activities for path 4 using the Nexus 6.
|
|
%Despite a short misdetection in seg. 2, caused by faulty pressure readings, the recognition can be considered to be very robust and accurate.
|
|
|
|
%Fixed Interval Smoothing
|
|
%
|
|
\begin{figure}
|
|
\input{gfx/particles/particles}
|
|
\caption{Comparison between the filtered and FBS particle set. Both have identical positions and are recorded at the same time step on path 4. The black dot indicates the estimation using the weighted arithmetic mean of all particles. The different colours represent the current weight of a certain particle.}
|
|
\label{fig:particles}
|
|
\end{figure}
|
|
At first, both FBS and BS are compared in context of fixed-interval smoothing.
|
|
As a reminder, fixed-interval smoothers are using all observations until time $T$ and therefore run offline, after the filtering procedure is finished.
|
|
%Thus, we calculate only the positional error between estimation and ground truth, since temporal information are negligible.
|
|
%
|
|
In contrast to BS, the FBS is not able to improve the results using the weighted arithmetic mean for estimating the current position.
|
|
Fig. \ref{fig:particles} illustrates the filtered and smoothed particle set at a certain time step on path 4.
|
|
It can be seen that the estimated position (black dot) for filtering and FBS is identical although the particle weights are highly different.
|
|
To address this problem, we are extending the FBS by adding a resampling step.
|
|
Here, particles with large weights are likely to be drawn multiple times.
|
|
This focuses the computational resources of the FBS into regions with high probability mass and finally improves the estimation.
|
|
Since smoothing operates on known states, the danger of sample impoverishment is negligible.
|
|
|
|
|
|
We deployed a multinomial resampling step to every smoothing interval of the FBS.
|
|
Now, the positional average error along all 4 paths using the Nexus and the Galaxy could be improved from \SI{2.08}{\meter} to \SI{1.37}{\meter}.
|
|
The BS performs with an average error of \SI{2.21}{\meter} for filtering and \SI{1.51}{\meter} for smoothing.
|
|
The difference between both filtering steps is of course based upon the randomized behaviour of the respective probabilistic models.
|
|
It is interesting to note, that the positional error is very similar for both used smartphones, although the approximation error varies greatly.
|
|
Using the FBS, the Galaxy provides an average approximation error of \SI{4.03}{\meter} while filtering resulted in \SI{7.74}{\meter}.
|
|
In contrast, the Nexus 6 filters with an error of \SI{5.11}{\meter} and \SI{3.87}{\meter} for smoothing.
|
|
The BS has a similar improvement rate.
|
|
|
|
|
|
\begin{figure}
|
|
\centering
|
|
\input{gfx/eval/interval_path2_compare/path2_interval_compare}
|
|
\caption{a) Exemplary results for path 2 where BS (blue) and filtering (green) using 2500 particles and 500 sample realisations. b) A situation where smoothing provides a worse error in regard to the ground truth, but obviously a more realistic path.}
|
|
\label{fig:int_path2}
|
|
\end{figure}
|
|
%
|
|
Two visual examples of the smoothing outcome for path 2 are illustrated in fig. \ref{fig:int_path2}.
|
|
It can be clearly seen, how the smoothing compensates for the faulty detected floor changes using future knowledge.
|
|
Additionally, the initial error is reduced extremely, approximating the pedestrian's starting position down to a few centimetres.
|
|
In the context of reducing the error as far as possible, fig. \ref{fig:int_path2} b) is a very interesting example.
|
|
Here, the filter offers a lower approximation and positional error in regard to the ground truth.
|
|
However it is obvious that smoothing causes the estimation to behave more natural, due to the restrictive smoothing transition, instead of walking the supposed path.
|
|
This phenomena could be observed for both smoothers respectively.
|
|
|
|
\begin{figure}
|
|
\centering
|
|
\input{gfx/eval/lag_path4_comp/path4_lag_comp}
|
|
\caption{Estimation results on path 4 for the filter and both smoothers using fixed-lag smoothing with $\tau = 5$. For a better visualisation, the segments are divided using an outline of alternating grey levels. The corresponding segment-error can be seen in fig. \ref{fig:lag_error_path4}.}
|
|
\label{fig:lag_comp_path4}
|
|
\end{figure}
|
|
\begin{figure}
|
|
\centering
|
|
\input{gfx/eval/lag_path4_comp/error_timed}
|
|
\caption{%
|
|
Error development while walking along Path 4 using the Nexus 6.
|
|
Especially in segments including floor changes, the error is reduced visibly by using smoothing methods.
|
|
The black line denotes the activity detected during each time step.
|
|
}
|
|
\label{fig:lag_error_path4}
|
|
\end{figure}
|
|
%
|
|
At next, we discuss the advantages and disadvantages of utilising FBS and BS as fixed-lag smoother.
|
|
Compared to fixed-interval smoothing, timing errors are now of higher importance due to an interest on real-time localization.
|
|
Especially interesting in this context are small lags $\tau < 10$ considering filter updates near \SI{500}{\milli\second}.
|
|
Fig. \ref{fig:lag_comp_path4} illustrates the different estimations for path 4 using a fixed-lag $\tau = 5$.
|
|
The associated approximation errors alongside the path can additionally be seen in fig. \ref{fig:lag_error_path4}.
|
|
Due to the small number of sample realisations for BS and the additional resampling for FBS, the errors are changing very frequently in contrast to the filter.
|
|
For better distinction, the path was divided into $10$ individual segments.
|
|
Again it can be observed, that both smoothers enable a better overall estimation especially in areas where the user is changing floors (cf. fig. \ref{fig:lag_error_path4} seg. 4, 7).
|
|
The black line denotes the activity detect during each time step.
|
|
Here, the camel humps while changing a floor can be explained by a flat area in the middle of the stairs.
|
|
Further, two short false detections can be observed in seg. 2 and seg. 6.
|
|
Immediately after the first floor change, a long and straight walk down the hallway follows.
|
|
While the Wi-Fi component pulls the pedestrian into the rooms on the right side, the actual walking route was located on the left side of the floor (see ground truth in fig. \ref{fig:lag_comp_path4} seg. 6).
|
|
Here, the BS is able to slightly improve the path, whereas the FBS follows the filtering until the upcoming staircase provides the necessary information for adjustments.
|
|
%It follows a critical area with high errors and multimodalities.
|
|
%Due to an in-house exhibition during the time of recording, we had to leave the ground truth by a few meters and Wi-Fi was strongly attenuated.
|
|
By looking at fig. \ref{fig:lag_comp_path4} seg. 9 it seems that both smoothing methods are highly improving the error.
|
|
However, the approximation error in this area is similar to the filter and only the positional error decreases.
|
|
This temporal error is caused by a phenomenon we call Wi-Fi jump.
|
|
Especially in seg. 8 and 9 a big crowd was gathered and highly attenuated the Wi-Fi signal.
|
|
For an excessive amount of time, the absolute location estimated by the Wi-Fi component got stuck in the middle of seg. 8 and therefore delayed the estimation.
|
|
The next viable measurements were then provided at the end of seg. 9.
|
|
This suggests that the here presented smoothing transition is able to improve the estimated path visibly, but does not compensate for those jumps in a temporal manner.
|
|
Finally, the BS provides an approximation error alongside all paths of $\SI{6.48}{\meter}$ for the Galaxy and $\SI{4.47}{\meter}$ for the Nexus, while filtering resulted in $\SI{7.92}{\meter}$ and $\SI{5.50}{\meter}$ respectively.
|
|
Whereas FBS improves the Galaxy's estimation from $\SI{7.73}{\meter}$ to $\SI{6.68}{\meter}$ and from $\SI{5.66}{\meter}$ to $\SI{4.80}{\meter}$ for the Nexus.
|
|
As stated before, the main advantage of BS over FBS is the better computational time by just using a sub-set of particles for calculations.
|
|
|
|
Reducing the number of particles down to $500$ does not necessarily worsen the estimation.
|
|
In most cases smoothing compensates for this reduction and maintains the good results.
|
|
Besides changing the number of particles, it is also possible to the variate the lag.
|
|
As one would expect, increasing the lag causes the smoothed estimation to approach the results provided by fixed-interval smoothing.
|
|
%This can be verified by looking at fig. \ref{}, which is a detailed view of segment XX of path 4 (cf. fig. \ref{fig:intcomp}).
|
|
It is obvious that a lag of \SI{30}{} time steps has access to much more future observations and is therefore able to obtain such a result.
|
|
Considering an update interval of \SI{500}{\milli\second}, a lag of \SI{30}{} would however mean that the smoother is \SI{15}{\second} behind the filter.
|
|
Nevertheless, there are practical applications like accurately verifying hit checkpoints or continuously optimizing a recurring segment of the path.
|
|
|
|
|
|
|