197 lines
16 KiB
TeX
197 lines
16 KiB
TeX
\section{Experiments}
|
|
|
|
% all paths we evaluated
|
|
\begin{figure}
|
|
\input{gfx/eval/paths}
|
|
\caption{The four paths that were part of the evaluation.
|
|
Starting positions are marked with black circles.
|
|
For a better visualisation they were slightly shifted to avoid overlapping.}
|
|
%\commentByFrank{font war korrekt, aber die groesse war zu gross im vgl. zu den anderen}
|
|
\label{fig:paths}
|
|
\end{figure}
|
|
%
|
|
The experiments were carried out on all floors (0 to 3) of the faculty building.
|
|
Each floor is about \SI{77}{\meter} x \SI{55}{\meter} in size, with a ceiling height of \SI{3}{\meter}.
|
|
To resemble real-world conditions, the evaluation took place during an in-house exhibition.
|
|
Thus, many places were crowded and Wi-Fi signals attenuated.
|
|
As can be seen in fig. \ref{fig:paths} we arranged 4 distinct walks, covering different distances, critical sections and uncertain decisions leading to multimodalities.
|
|
The ground truth is measured by recording a timestamp at marked spots on the walking route.
|
|
When passing a marker, the pedestrian clicked a button on the smartphone application.
|
|
Between two consecutive points, a constant movement speed is assumed.
|
|
Thus, the ground truth might not be \SI{100}{\percent} accurate, but fair enough for error measurements.
|
|
The approximation error is then calculated by comparing the interpolated ground truth position with the current estimation.
|
|
Especially in the context of smoothing, it is also very interesting to exclude the temporal delay from the error calculations and measure only the positional difference between estimated and ground truth path.
|
|
This gives a statement about the extent to which the smoothed path superficially improves compared to the filtered one.
|
|
|
|
All walks start with a uniform distribution (random position and heading) as prior for $\mStateVec_0$.
|
|
To allow the system to stabilize its initial state, the first few estimations are omitted from error calculations.
|
|
Even though, the error during the following few seconds is expected to be much higher than the error when starting with a well known initial position and heading.
|
|
|
|
The measurements were recorded using a Motorola Nexus 6 and a Samsung Galaxy S5.
|
|
As the Galaxy's \docWIFI{} can not be limited to the \SI{2.4}{\giga\hertz} band only, its scans take much longer than those of the Nexus: \SI{3500}{\milli\second} vs. \SI{600}{\milli\second}.
|
|
Additionally, the Galaxy's barometer sensor provides fare more inaccurate and less frequent readings than the Nexus does.
|
|
This results in a better localisation using the Nexus smartphone.
|
|
The computation for both filtering and smoothing was done offline using the aforementioned \mbox{CONDENSATION} algorithm and multinomal (cumulative) resampling.
|
|
%However, the filter itself would be fast enough to run on the smartphone itself ($ \approx \SI{100}{\milli\second} $ per transition, single-core Intel\textsuperscript{\textregistered} Atom{\texttrademark} C2750).
|
|
%The computational times of the different smoothing algorithm will be discussed later.
|
|
Unless explicitly stated, the state was estimated using the weighted arithmetic mean of the particles.
|
|
|
|
As mentioned earlier, the position of all \docAP{}s (about 5 per floor) is known beforehand.
|
|
Due to legal terms, we are not allowed to depict their positions and therefore omit this information within the figures.
|
|
Additionally, we used five \docIBeacon{}s for slight enhancements in some areas.
|
|
The empirically chosen values for \docWIFI{} were $P_{0_{\text{wifi}}} = \SI{-46}{\dBm}, \mPLE_{\text{wifi}} = \SI{2.7}{}, \mWAF_{\text{wifi}} = \SI{8}{\dB}$, and $\mPLE_{\text{ib}} = \SI{1.5}{}$ for the \docIBeacon{}s, respectively.
|
|
The system was tested by omitting any time-consuming calibration processes for those values.
|
|
We therefore expect the localisation process to perform generally worse compared to standard fingerprinting methods \cite{Ville09}.
|
|
However, smoothing will often compensate for those poorly chosen system parameters.
|
|
|
|
For the filtering we used $\sigma_\text{wifi} = \sigma_\text{ib} = 8.0$ as uncertainties, both growing with each measurement's age.
|
|
While the pressure change was assumed to be \SI{0.105}{$\frac{\text{\hpa}}{\text{\meter}}$}, all other barometer-parameters are determined automatically (see \ref{sec:eval}).
|
|
The step size $\mStepSize$ for the transition was configured to be \SI{70}{\centimeter} with an allowed derivation of \SI{10}{\percent}. The heading deviation was set to \SI{25}{\degree}.
|
|
Edges departing from the pedestrian's destination are downvoted using $\mUsePath = 0.9$.
|
|
For smoothing we set $\sigma^2_{\text{turn}} = \SI{5}{\degree}$ and $\sigma^2_z = \SI{0.25}{\centimeter}$.
|
|
If walking or unknown are the current activities, $ \mu_{\text{step}} = \SI{0.7}{\meter}$, $ \mu_{\text{step}} = \SI{0.5}{}$ and $ \mu_z = \SI{0.0}{\meter}$ are used.
|
|
Walking upstairs sets $ \mu_{\text{step}} = \SI{0.4}{\meter}$, $ \sigma_{\text{step}}^2 = \SI{0.2}{}$ and $ \mu_z = \SI{-0.3}{\meter}$, otherwise $ \mu_{\text{step}} = \SI{0.5}{\meter}$, $ \sigma_{\text{step}}^2 = \SI{0.3}{}$ and $ \mu_z = \SI{0.3}{\meter}$ for walking downstairs.
|
|
|
|
|
|
%kurz zeigen das activity recognition was bringt
|
|
\begin{figure}
|
|
\input{gfx/activity/activity_over_time}
|
|
\caption{activity recognition}
|
|
\label{fig:activityRecognition}
|
|
\end{figure}
|
|
By adding the activity recognition to the system of \cite{Ebner-16}, we are able to further improve the overall localisation results.
|
|
The approximation error decreases by an average of \SI{XX}{\centimeter} for all 4 paths on 10 MC runs.
|
|
Due to this additional knowledge, the state transition samples mostly depending upon the current activity and therefore limits the possibility of false floor changes.
|
|
Fig. \ref{fig:activityRecognition} shows the recognized activities for path 4 using the Nexus 6.
|
|
Despite a short misdetection in seg. 2, caused by faulty pressure readings, the recognition can be considered to be very robust and accurate.
|
|
|
|
%Fixed Interval Smoothing
|
|
At first, both FBS and BS are compared in context of fixed-interval smoothing.
|
|
As a reminder, fixed-interval smoother are using all observations until time $T$ and therefore run offline, after the filtering procedure is finished.
|
|
Thus, we calculate only the positional error between estimation and ground truth, since timely information are negligible.
|
|
%
|
|
\begin{figure}
|
|
\input{gfx/particles/particles}
|
|
\caption{particles. green = avg50, black = avg. gnuplot zickt bei der legende}
|
|
\label{fig:particles}
|
|
\end{figure}
|
|
In contrast to BS, the FBS is not able to improve the results using the weighted arithmetic mean for estimating the current position.
|
|
Fig. \ref{fig:particles} illustrates the filtered and smoothed particle set at a certain time step on path 4.
|
|
It can be seen that the estimated position (black dot) for filtering and FBS is identical although the particle weights are highly different.
|
|
To address this problem, we are extending the FBS by adding a resampling step.
|
|
Here, particles with large weights are likely to be drawn multiple times.
|
|
This focuses the computational resources of the FBS into regions with high probability mass and finally improves the estimation.
|
|
Since smoothing operates on known states, the danger of sample impoverishment is negligible.
|
|
|
|
|
|
We deployed 10 MC runs using \SI{2500}{} particles for approximation and a multinomial resampling step to every smoothing interval of the FBS.
|
|
Now, the positional average error along all 4 paths using the Nexus and the Galaxy could be improved from \SI{2.08}{\meter} to \SI{1.37}{\meter}.
|
|
Using the same number of particles and $500$ sample realisations the BS performs with an average error of \SI{2.21}{\meter} for filtering and \SI{1.51}{\meter} for smoothing.
|
|
The difference between both filtering steps is of course based upon the randomized behaviour of the respective probabilistic models.
|
|
It is interesting to note, that the positional error is very similar for both used smartphones, although the approximation error varies greatly.
|
|
Using the FBS, the Galaxy donates an average approximation error of \SI{4.03}{\meter} by filtering with \SI{7.74}{\meter}.
|
|
In contrast the Nexus 6 filters at \SI{5.11}{\meter} and results in \SI{3.87}{\meter} for smoothing.
|
|
The BS has a similar improvement rate.
|
|
|
|
A visual example of the smoothing outcome for path 3 is illustrated in fig. \ref{fig:int_path3_a}.
|
|
It can be clearly seen, how the smoothing compensates for the faulty detected floor change using future knowledge.
|
|
|
|
|
|
The estimation of BS looks way more realistic and adapts better to the ground truth path.
|
|
%However, in this particular example the FBS starts at an earlier position (cf. fig. \ref{fig:intcomp} seg. 2), better handling the initial uniform distribution.
|
|
%Another advantage of BS over FBS, is the ability to still improve the filtering results even while reducing the number of particles radical.
|
|
%For example \SI{50}{} particles and \SI{25}{} sample realisations are providing reliable estimations similar to above experiments, though the risk of losing track is higher.
|
|
|
|
\begin{figure}
|
|
%\begin{subfigure}{0.175\textwidth}
|
|
% \input{gfx/eval/interval_path2_good/path2_interval_good}
|
|
% \caption{}
|
|
% \label{fig:int_path2_a}
|
|
%\end{subfigure}
|
|
%\begin{subfigure}{0.175\textwidth}
|
|
% \input{gfx/eval/interval_path2_bad/path2_interval}
|
|
% \caption{}
|
|
% \label{fig:int_path2_b}
|
|
%\end{subfigure}
|
|
%\caption{a) Exemplary results for path 2 where BS (blue) and filtering (green) using 2500 particles and 500 sample realisations. b) A situation where BS smoothing was not able to improve the filtering results. Two main factors are causing this: an initial position within a detached room and inaccurate pressure readings given by the Galaxy S5.}
|
|
\input{gfx/eval/interval_path2_compare/path2_interval_compare}
|
|
\caption{left) Exemplary results for path 2 where BS (blue) and filtering (green) using 2500 particles and 500 sample realisations. right) A situation where BS smoothing was not able to improve the filtering results. Two main factors are causing this: an initial position within a detached room and inaccurate pressure readings given by the Galaxy S5.}
|
|
\label{fig:int_path2_comp}
|
|
\end{figure}
|
|
Despite the very good outcomes provided by both interval smoother, there are some rare situations in which smoothing does not improve the filtered estimation or even improves the visual path.
|
|
For example fig. \ref{fig:int_path3_b} depicts such a situation for path 3 using BS and measurements provided by the Galaxy S5.
|
|
Here, the estimation was not able to change floors correctly due to faulty pressure readings. Additionally, the initial position was located within a detached room.
|
|
This shows that the smoothing results are of course highly depend upon the filtering performance.
|
|
|
|
At next, we discuss the advantages and disadvantages of utilizing FBS and BS as fixed-lag smoother.
|
|
Compared to fixed-interval smoothing, timely errors are now of higher importance due to an interest on real-time localization.
|
|
Especially interesting in this context are small lags $\tau < 10$ considering filter updates near \SI{500}{\milli\second}.
|
|
|
|
%Path 4 grafik mit fixed-lags
|
|
|
|
%
|
|
%wie gut ist fixed-lag mit einem lag = 5. was fällt so auf?
|
|
Fig. \ref{fig:lag_error_path4} illustrates the different approximation errors alongside path 4 using $500$ particles, \SI{100}{sample realisations} for BS and a fixed-lag $\tau = 5$.
|
|
%
|
|
\begin{figure}
|
|
\input{gfx/eval/lag_path4_error/error_timed_costum}
|
|
\caption{Hier muss ich noch was schreiben. Bin aber seeeehr muede. Geh jetzt ins Bett. Nachti.}
|
|
\label{fig:lag_error_path4}
|
|
\end{figure}
|
|
%
|
|
Due to the small number of sample realisations for BS and the different estimation method for FBS, the errors are changing very frequently in contrast to the filter.
|
|
It can be seen that again BS provides a better overall estimation, especially in areas where the user is changing floors (cf. fig. \ref{fig:lag_error_path4} seg. 4, 9).
|
|
Besides the positional quality, also the timely error could be reduced by both algorithms in fig. \ref{fig:lag_error_path4}.
|
|
Once more, the BS outperforms the FBS by providing an overall approximation error of $\SI{4.86}{\meter}$ for the Galaxy and $\SI{2.97}{\meter}$ for the Nexus by filtering with $\SI{5.68}{\meter}$ and $\SI{3.15}{\meter}$ respectively.
|
|
Whereas FBS is not able to reduce the filtering error, obtaining $\SI{5.59}{\meter}$ using the Nexus and $\SI{3.12}{\meter}$ the Galaxy.
|
|
However, this does not mean, that the FBS is unpracticable for problems of fixed-lag smoothing.
|
|
By looking at the data in detail, high errors similar as seen in fig. \ref{fig:lag_error_path4} occur on path 3, which cause the estimation to behave more natural instead of walking the supposed path.
|
|
This phenomena could be observed for both smoothers respectively.
|
|
Also note the difference between this error, including timely information, and the positional error used before.
|
|
%The median errors for all conducted walks are listed in table \ref{}.
|
|
Similar to fixed-interval smoothing, decreasing the number of particles does not necessarily worsen the estimation.
|
|
In most cases smoothing compensates for this reduction and maintains the good results.
|
|
%For example estimating path for using \SI{50}{particles} results in an approximation error for BS of \SI{}{\meter}.
|
|
|
|
%lag vergrößern was passiert beschreiben
|
|
Besides changing the number of particles, it is also possible the variate the lag.
|
|
As one would expect, increasing the lag causes the smoothed estimation to approach the results provided by fixed-interval smoothing.
|
|
This can be verified by looking at fig. \ref{}, which is a detailed view of segment XX of path 4 (cf. fig. \ref{fig:intcomp}).
|
|
It is obvious that a lag of \SI{30}{} time steps has access to much more future observations and is therefore able to obtain such a result.
|
|
Considering an update interval of \SI{500}{\milli\second}, a lag of \SI{30}{} would however mean that the smoother is \SI{15}{\second} behind the filter.
|
|
Nevertheless, there are practical applications like accurately verifying hit checkpoints or continuously optimizing a recurring segment of the path.
|
|
|
|
%verlgeich noch zwischen bs und fbs. bs weniger partikel und somit schneller. ändern der estimations kann aber auch ein möglichkeit sein.
|
|
|
|
|
|
|
|
|
|
%fixed-lag reduces the error about... however, as seen in fig. \ref{} ist der bloße error nicht unbedingt ausschlaggebend für die verbesserung. fast immer liefert smoothing pfade die realistischer sind, aber die error erhöhen.
|
|
|
|
%conclusion der experimente
|
|
%bei weniger partikeln bringt fixed-lag und fixed-interval smoothing im verhältnis sogar mehr! weil es da mehr zum "aufräumen" gibt. trotzdem hängt die performane natürlich stark vom vorwärtsschritt ab und man sollte nicht all zu wenige waehlen und lieber auf nummer sicher gehen. bs ist im vergleich zu fbs ein gutes stück besser in unserem fall. das hängt auch stark mit dem bereits sehr guten filtering schritt zusammen. man könnte aber trotzdem schlussfolgern das bs besser für indoor ist.
|
|
|
|
%fbs ist hier mies und liegt direkt über dem filter? oder ein fbs mit anderer estimaton und den filter nicht anzeigen?! das ist doch quatsch... wennn ich den filter net anzeige. notfalls einfach den fbs nicht nehmen. sondern sagen das er net taugt und rauswerfen.
|
|
|
|
%Smoothing mit großen lag kann die zeitliche information schwer halten. das liegt hauptsächlich daran, das im smoothing nur die relativen positionsinfos genutzt werden. das wi-fi wird nicht beachtet und deswegen können absolute justierungen der position (sprünge) nur sehr schlecht abgefedert werden.
|
|
|
|
|
|
%fixed lag smoothing verbessert den pfad ein wenig und vor allem bügelt unrealistische sprunge aus. daher wird der zeitlich verzug gut herrausgerechnet.
|
|
|
|
%beispiel multimodalität
|
|
|
|
%Tabelle mit spalten interval partikel, lag, partikel; spalten: filter, BS, FBS
|
|
|
|
%Evaluation:
|
|
%\begin{itemize}
|
|
% \item Filter ist immer der gleiche mit MultiPathPrediction und Importance Factors
|
|
% \item FBS Interval mit 500 und 2500 Partikeln auf 4 Pfaden mit SimpleSmoothingTrans
|
|
% \item BS Interval mit 500 zu 100 und 2500 zu 500 Partikeln auf auf 4 Pfaden mit SimpleSmoothingTrans
|
|
% \item FBS Lag = 5 mit 500 und 2500 Partikeln auf 4 Pfaden mit SimpleSmoothingTrans
|
|
% \item BS Lag = 5 mit 500 zu 100 und 2500 zu 500 Partikeln auf auf 4 Pfaden mit SimpleSmoothingTrans
|
|
% \item BS Lag zu Error Plot. Lag von 0 bis 100, wie verhält sich der Error. Am besten auf Pfad 4 mit SimpleSmoothingTrans.
|
|
%\end{itemize}
|
|
|
|
|
|
|