\section{Experiments}

% allgemeine infos über pfade und gebäude. wo 

%Gebäude
All upcoming experiments were carried out on four floors (0 to 3) of a \SI{77}{m} x \SI{55}{m} sized faculty building. 
It includes several staircases and elevators and has a ceiling height of about \SI{3}{m}. 
The gridded graph was generated for the complete campus and thus outdoor areas like the courtyard are also walkable.
As \docWIFI{} is attenuated by obstacles and walls, it does not provide a consistent quality over the complete building. 
To get an idea about the quality, we interpolated the \docWIFI{} quality factor $q(\mObsVec_t^{\mRssiVec_\text{wifi}})$, with $l_{\text{max}} = \SI{-75}{\dBm}$ and $l_{\text{min}} = \SI{-90}{\dBm}$, between some measurements using a nearest-neighbour procedure.
In fig. \ref{fig:paths} the resulting colourized floorplan is shown.
Here, the intensity of red indicates a low signal strength and thus a bad quality for localising a pedestrian within this area using \docWIFI{}.

% bild: mit pfaden drauf und eventl. wifi qualität in jeweiligen bereichen? (kann frank das)
	\begin{figure}
		\centering
		\input{gfx/eval/paths.tex}
		\caption{The three paths that were part of the experiments. Starting positions are marked with black circles. The red squares illustrate the \docWIFI{} quality in this sector. The intensity of red indicates a low coverage and thus a bad quality for localisation.}
		\label{fig:paths}
	\end{figure}
%
%Pfade
We arranged three distinct walks (see also fig. \ref{fig:paths}).
The measurements for the walks were recorded using a Motorola Nexus 6 at 2.4 GHz band only. 
The computation was done offline as described in algorithm \ref{fig:paths}. 
For each walk we deployed $50$ runs using 5000 particles for each mode.
Instead of an initial position and heading, all walks start with a uniform distribution (random position and heading) as prior $q_1$.
In the graph-based transition of the dominant filter, the to-be-walked distance is given by the number of steps using a step size of \SI{70}{\centimeter} with an allowed deviation of \SI{10}{\percent}. 
The deviation for the walking direction was set to \SI{25}{\degree}.
Since the simple transition randomly scatters particles within a specific range, we choose a covariance matrix that allows a variance of \SI{200}{\centimeter} in $x$- and $y$-direction for the multivariate normal distribution. 
Here, floors were changed by deploying a discrete distribution for every floor level, providing a chance of \SI{27}{\percent} for changing one floor and \SI{5}{\percent} for two floors in a particular $z$-direction.

We omit any time-consuming calibration processes and therefore use the same parameters for all \docWIFI{} access-points, similar to \cite{Ebner-15}. 
The position of the access-points (about five per floor) is known beforehand. 
Due to legal terms, we are not allowed to depict their positions and therefore omit this information within the figures.
To model the uncertainty of the \docWIFI{}, a white Gaussian noise with $\sigma_\text{wifi} = \SI{8}{\dBm}$, growing with each measurement's age, was chosen.
The pressure change was assumed to be \SI{0.105}{$\frac{\text{\hpa}}{\text{\meter}}$}, all other barometer-parameters are determined automatically as described in \cite{Fetzer2016OMC}. 

As expected before, the kernel density estimation in \eqref{eq:KDE} and the much simpler multivariate normal distribution estimation are both producing similar results in context of the divergence measure given in \eqref{equ:KLD}. 
For this reason and because of the faster processing time, especially in context of smartphones, we estimated the posteriors of the modes by approximating a multivariate normal distribution (cf. section \ref{sec:divergence}).
For calculating \eqref{equ:KLD} we set $\lambda = 0.03$ based on best practices. 
Finally, the pedestrian's most likely position (state) was then estimated using the weighted arithmetic mean of the particle set.

% ground truth
The ground truth is measured by recording a timestamp at marked spots on the walking route. When passing a marker, the pedestrian clicked a button on the smartphone application. 
Between two consecutive points, a constant movement speed is assumed.
Thus, the ground truth might not be \SI{100}{\percent} accurate, but fair enough for error measurements. 
The approximation error is then calculated by comparing the interpolated ground truth position with
the current estimation \cite{Fetzer2016OMC}.

The here presented walks were selected because they fail in some way using a restrictive transition model and thus are well suited to represent the benefits and drawbacks of the IMMPF approach.
In this context, it should be noted that the localisation system used for the experiments is very basic and can be seen as a slimmed version of our previous works \cite{Fetzer2016OMC, Ebner-16}. 
Optimizing the \docWIFI{} parameters and adding additional methods will improve the localisation results significantly. 

To avoid misunderstandings in the upcoming discussions, the used terminologies for the different filter schemes are summarized as follows: 
graph-based and simple filter are standard independently running particle filters (PF) using the graph-based and simple transition as described at the end of section \ref{sec:rse}. 
In contrast, dominant and support filter are indeed the same filters, but used as modes within the IMMPF procedure as described in section \ref{sec:immpf}. 
To allow a detailed discussion of the results shown by images, we separated the paths in different segments (seg.) of interest.

%error at the beginning always very high. about 44 meters. therefore the median is better value oder 75 quantil.

% zeigen das es stucken verhindert (eventl. hier eine andere aufnahme die mitten drinnen stecken bleibt)
% bild: stucken im raum + nicht mehr stucken im raum + kld mit anzeigen
	\begin{figure}[b]
		\centering
		\input{gfx/eval/path3.tex}
		\input{gfx/eval/path3-kld.tex}
		\caption{Exemplary results on path 3 for the graph-based filter (red), the simple filter (blue) and our IMMPF approach (green). The Kullback-Leibler divergence $D_{\text{KL}}$ between the graph-based and the simple filter (purple) proves itself as a good indicator, if one filter gets stuck or loses track.}
		\label{fig:path3}
	\end{figure}
	%
First, we discuss the results of path 3, starting at the left-hand side of the building. 
Exemplary estimation results for the IMMPF, the graph-based and the simple particle filter, can be seen in fig. \ref{fig:path3}. 
As mentioned above, every run of a walk starts with a uniform distribution as prior.
Due to a low \docWIFI{} coverage at the starting point in seg. 1, the pedestrian's position is falsely estimated into a room instead of the corridor.   
All three filters are able to overcome this false detection. 
However, the graph-based filter (red) gets then indissoluble captured within a room, because of its restrictive behaviour and the aftereffects of the initial \docWIFI{} readings (cf. fig. \ref{fig:path3} seg. 2). 
It provides a \SI{75}{\percent}-quantil of $\tilde{x}_{75} = \SI{3884}{\centimeter}$ and got stuck in \SI{100}{\percent} of all runs.
As expected and discussed earlier, the simple filter (blue) is less prone to bad observations and provides not accurate, but very robust results of $\tilde{x}_{75}= \SI{809}{\centimeter}$ and a standard deviation over all results of $\bar{\sigma} = \SI{529}{\centimeter}$.
Looking at $D_{\text{KL}}$ over time confirms our assumption made in section \ref{sec:divergence}. 
The graph-based filter (red) gets stuck and is not able to recover. 
Starting on from this point, the Kullback-Leibler divergence $D_{\text{KL}}$ (purple) further increases due to the growing distance between both filters (blue and red) starting at seg. 2 until the end of seg. 4.
It is clearly visible, that this divergence between both filters is a very good indicator to observe, if a filter gets stuck or loses track.
Following, the IMMPF (green) results in a very natural and straight path estimation and a low $D_{\text{KL}}$ between modes.
Of course, the low $D_{\text{KL}}$ of the IMMPF results from the mixing between the modes caused by $\Pi_t$.
Therefore, the matrix $\Pi_t$ can also be seen as some kind of limiter, paying attention that the modes do not diverge to much from each other.
The benefits of mixing both filtering schemes within the scenario of path 3 are thus obvious.
The IMMPF filters with an error of $\tilde{x}_{75} = \SI{667}{\centimeter}$ and $\bar{\sigma} = \SI{558}{\centimeter}$.

% zeigen das schlechtes wi-fi (zu hohe diversity) behoben wird. 
% bild: lauf auf der rechten seite des gebäudes zeige mit und ohne wifi faktor (schlechtes wifi einzeichnen)
	\begin{figure}[b]
		\centering
		\input{gfx/eval/path2.tex}
		\input{gfx/eval/path2-wifi-quality.tex}
		\caption{Comparison of the estimation results on path 2 with (green) and without (red) the \docWIFI{} quality factor in the Markov transition matrix. The low \docWIFI{} quality and thus high errors between the \SI{80}{th} and \SI{130}{th} second are caused by the high attenuation and low signal coverage inside the zig-zag stairwell on the building's backside.}
		\label{fig:path2}
	\end{figure}
	%
Next, we investigate the performance of our approach by considering the scenario in path 2. 
Here, the overall \docWIFI{} quality is rather low, especially in the zig-zag stairwell on the building's back and the small entrance area at floor 1 (cf. fig. \ref{fig:paths} and fig. \ref{fig:path2} seg. 3). 
Path 2 starts in the second floor, walking down the centred stairs into the first floor, then making a right turn and walking the stairs down to zeroth floor. From there we walk back to the second floor using the zig-zag stairwell and after finally crossing a room we are back at the start. 
This is a very challenging scenario, at first the estimation got stuck on the first floor in a room's corner for \SI{20}{\second} (see fig. \ref{fig:path2} seg. 2) and after that the \docWIFI{} is highly attenuated at the beginning and end of seg. 3.
Looking at fig. \ref{fig:path2}, one can observe the impact of the \docWIFI{} quality factor within the Markov transition matrix. 
Without it, the position estimation (red) is drifting into the courtyard, missing the stairwell and producing high errors between \SIrange{80}{130}{\second} in seg. 3. 
As described earlier, the bad \docWIFI{} readings are causing $D_{\text{KL}}$ to grow.
It follows that the accurate dominant filter draws new particles from the uncertain support and therefore worsens the position estimation.
Given this outcome, it is not very satisfying that the system is able to recover thanks to its high diversity during situations with uncertain measurements.
By adding the \docWIFI{} quality factor, the system is able to improve the approximated path (green) and the overall estimation results from $\tilde{x}_{75} = \SI{1278}{\centimeter}$ with $\bar{\sigma} = \SI{948}{\centimeter}$ to $\tilde{x}_{75} = \SI{953}{\centimeter}$ with $\bar{\sigma} = \SI{543}{\centimeter}$.
However, this is far from perfect and in some cases ($\sim \SI{9}{\percent}$) the quality factor was not able to prevent the estimation from drifting into the courtyard. 
This solely happened when particles were sampled directly onto the courtyard while changing from first to zeroth floor (cf. fig. \ref{fig:path2} seg. 3), instead of within the building first. 
Those particles then received a high weight due to the attenuated measurements, causing a weight degeneracy.
Adapting the bounds $l_{\text{max}}$ and $l_{\text{min}}$ of the quality factor or optimizing the access-point's parameters can resolve this problem \cite{Ebner-17}.

% zeigen das immpf nicht viel schlechter als normaler pf (ohne stucken) ist. 
% bild: er schafft es nicht die treppe rauf + er schafft es immpf + er schafft es normal filter
	\begin{figure}[t]
		\centering
		\input{gfx/eval/path1.tex}
		\input{gfx/eval/path1-time.tex}
		\caption{Estimation results and error development while walking alongside path 1. In \SI{20}{\percent} of cases, the the graph-based particle filter failed to detect the first floor change. Therefore, a good (blue) and a bad (red) result are shown. The here presented approach (green) never lost track.}
		\label{fig:path1}
	\end{figure}
	%
An exemplary result for path 1 is illustrated in fig. \ref{fig:path1}. 
The path starts on the first floor and finishes on the third after walking two straight stairs. 
Using the graph-based filter for localisation, we were able to obtain a very accurate path (blue) in \SI{80}{\percent} of the runs providing $\tilde{x}_{75} = \SI{526}{\centimeter}$ with $\bar{\sigma} = \SI{316}{\centimeter}$. 
Due to a lack of particles near the stairs, the other \SI{20}{\percent} of the estimations given by the graph-based filter failed to detect the first floor change (red). 
Using our approach (green), we were able to detect all floor changes and thus never lost track. 
Fig. \ref{fig:path1} shows an example, where the dominant filter also failed to change floors within seg. 2. 
By looking at this segment of the error plot, we can observe a more or less constant error of \SIrange{5}{6}{\meter}, which then drops rapidly at the beginning of seg. 3. 
This is due to the growing importance of the mixing stage, where more and more particles from the support filter are incorporated into the dominant filter.
Going on in seg. 3, the \docWIFI{} measurements suffer from an attenuation directly after leaving the stairs, what leads to a high error using the graph-based filter (blue and red). 
In contrast, the IMMPF is able to compensate the false detection due to an decreasing \docWIFI{} quality and thus a highly focused posterior provided by the dominant filter. 
The IMMPF then performs with $\tilde{x}_{75} = \SI{544}{\centimeter}$ and $\bar{\sigma} = \SI{281}{\centimeter}$. 
Those very similar estimation results between IMMPF (green) and the graph-based filter (blue) confirm the efficiency of the mixing and how it is able to keep the accuracy while providing a higher robustness against failures.

%allgemeines abschließendes blabla?
To summarize, the here presented approach was able to recover in all situations and thus never got stuck within a demarcated area.
The above deployed experiments have shown, that the Markov switching process, as presented in sec. \ref{sec:immpf}, enables a reasonable mixing between two particle filters with different transition schemes.
Therefore, the quality and success of the results depend highly on the parameters used within the Markov matrix $\Pi_t$. 
Those values are very sensitive and should be chosen carefully in regard to the specific system and use case.
The experiments have further shown, that faulty \docWIFI{} measurements can lead to significant errors in a very short time.
That has to do with the fact that \docWIFI{} is the only absolute information source within the state evaluation and thus plays a significant part in the mixing stage of \eqref{equ:immpMode2}.
Adding additional absolute sensors like bluetooth beacons or ultrasonics sensors, it is possible to reduce the \docWIFI{} component's significance towards other sensor models evaluating the pedestrian's possible whereabouts.


% gegenüberstellung aller pfade und werte in tabelle
%	\begin{table}
%		\caption{Resulting Errors for all conducted walks in meter.}
%		\label{tbl:err}
%		\centering
%		\scalebox{0.93}{
%		\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|}
%			\hline
%			& \multicolumn{3}{c|}{Path 1} & \multicolumn{3}{c|}{Path 2} & \multicolumn{3}{c|}{Path 3}\\
%			\hline
%			 &  $\bar{x}$ & $\bar{\sigma}$ & $\tilde{x}_{75}$ & $\bar{x}$ & $\bar{\sigma}$ & $\tilde{x}_{75}$ & $\bar{x}$ & $\bar{\sigma}$ & $\tilde{x}_{75}$ \\
%			\hline
%			 $\text{PF}_{\text{graph}}$ & $4.0$ & $3.2$ & $5.3$ & $8.2$ & $4.0$ & $10.7$ & $30.3$ & $12.8$ & $38.8$  \\
%			\hline
%			 $\text{PF}_{\text{simple}}$ & $4.9$ & $2.8$ & $6.2$ & $7.3$ & $2.9$ & $9.4$ & $6.8$ & $5.4$ & $8.1$ \\
%			\hline
%			 IMMPF & $4.2$ & $2.8$ & $5.4$ & $7.7$ & $5.4$ & $9.5$ & $6.3$ & $5.6$ & $6.7$ \\
%			\hline
%		\end{tabular}
%		}
%	\end{table}

%An overview of all localisation results can be seen in table \ref{tbl:err}.

%So the results of table \ref{tbl:err} should not be seen as best case localization results, but more as proofing robustness while providing room for further improvements.

%Especially, the graph-based transition model allows many optimizations and performance boosts.
%More importantly, the here presented approach was able to recover in all situations and thus never got completely stuck within a demarcated area. 
%All results were similar or more accurate then the ones provided by the standalone filters, even when the localisation did not suffer from any problems.

%how the Markov transition matrix regulates the impact of the supporting filter in the right amount.




