\section{Experiments}

As explained at the very beginning of this work, we wanted to explore the limits of the here presented localization system.
By utilizing \del{it to} \add{the proposed technology in} a 13th century historic building, we created a challenging scenario not only because of the various architectural factors, but also because of its function as a museum.
During all experiments, the museum was open to the public and had a varying number of \SI{10}{} to \SI{50}{} visitors while recording.

The \del{\SI{2500}{\square\meter}} building consists of \SI{6}{} different levels, which are grouped into 3 floors (see fig. \ref{fig:apfingerprint}). 
Thus, the ceiling height is not constant over one floor and varies between \SI{2.6}{\meter} to \SI{3.6}{\meter}.
While most of the exterior and ground level walls are made of massive stones, the floors above are half-timbered constructions.
In the middle of the building is an outdoor area, which is only accessible from one side. 
\add{The total walkable indoor area for a visitor is \SI{2500}{\square\meter} in size. 
Due to objects like exhibits, cabinets or signs not all positions within the building were freely accessible.}
For the sake of simplicity we did not incorporate such knowledge into the floor plan. 
Thus, the floor plan consists only of walls, ceilings, doors, windows and stairs.
It was created using our 3D map editor software (see fig. \ref{fig:mapeditor}) based on architectural drawings from the 1980s. 
\add{Our map editor is also used to automatically create the navigation mesh, which only takes a few seconds to compute.}

\begin{figure}[t]
		\centering
		\includegraphics[width=\textwidth]{gfx/apps/editor_light.png}
        \caption{\add{The 3D map editor we developed to create the floor plans. This screenshot shows the ground level of the building. The window is split into toolbar (left), layers (upper right), parameters of current selection (lower right), drawing mode (upper center) and 3D view (lower center).}}
		 \label{fig:mapeditor}
\end{figure}

%wie haben wir die ap aufgehängt 
\add{As described in section \ref{sec:wifi} we used \SI{42}{} WEMOS D1 mini to provide a \docWIFI{} signal coverage throughout the building.
The distribution of the beacons on ground floor can be seen in fig. \ref{fig:apfingerprint} (black dots) as well as the references (fingerprints) for optimization. 
The position of the beacons were chosen depending on available power sources. 
Care was taken to have at least two beacons in each room and a third beacon visible in an approximate radius of \SI{10}{\meter}.
Due to the difficult architecture and the extremely thick walls of the museum, we decided on this procedure, which explains the rather large number of \SI{42}{} transmitters compared to modern buildings. 
Another reason for the high number of beacons is that we did not want to analyze the quality of the Wi-Fi signal coverage for further improvements, as this can be a very time-consuming task. 
In many areas of the building an improvement would not even be possible due to the lack of power sockets. 
To compensate that, battery powered beacons could be used but we consider this approach less practicable, so we did not take this option. 
The power sockets are located at different heights ranging from \SI{0.2}{\meter} to \SI{2.5}{\meter}. 
Consequently, there were no prior requirements on how a single beacon should be placed exactly and its position is dictated by the socket's position. 
Considering all the above, the beacons were placed more or less freely and to the best of our knowledge.} 

\add{A similar approach was chosen for placing the fingerprints. 
The positions of the fingerprints are set within our 3D map editor (see fig. \ref{fig:mapeditor}) software by dragging the fingerprinting icon on the desired position or by entering the position manually. 
The reference points were placed every \SI{3}{\meter} to \SI{7}{\meter} from each other, however as can be seen in fig. \ref{fig:apfingerprint} not necessarily accurate.
As the optimization scheme does not require equally spaced reference points, doing so would result in superfluous effort.
Furthermore, it is not easy to adopt the exact position to take the reference measurements in the building later on. 
Of course, this could be achieved with appropriate hardware (e.g. laser-scanner), but again, this requires more time and care, which in our opinion does not justify a presumably increased accuracy of some decimeters.}

\add{Summing up the above, the following initial steps are required to utilize our localization system to a building: 
\begin{enumerate}
	\item Acquiring a blueprint or architectural drawing of the building including at minimum the walls and stairs of the respective floors. 
	\item Based on this 2D drawing, the floor plan is created manually using our 3D map editor (cf. fig. \ref{fig:mapeditor}), comparable to software like Inkscape or FreeCAD. 
	\item If necessary, create or improve the Wi-Fi infrastructure by plugging in beacons to available power sockets and compose a whitelist of MAC-addresses of the involved access points or beacons.
	%\item Store floor plan and whitelist of MAC-addresses onto the smartphone. 
	\item Record the reference measurements based on the reference positions given in the floor plan. 
	\item The Wi-Fi model is optimized using the previously obtained reference measurements.
	\item The navigation mesh is created automatically based on the before created floor plan as can be seen in fig. \ref{fig:museumMapMesh}. 
\end{enumerate}
For the building considered within this work, we were able to perform this steps in less then \SI{160}{\minute} by a person, which is familiar with the system, and the janitor of the museum. 
Step 1 and 2 were conducted off-site. 
The blueprint was initially provided by the director of the museum as digital photography. 
Creating the floor plan including walls and stairs took us approximately \SI{40}{\minute} and is then stored onto the smartphone after creation. 
Adding knowledge like semantic information such as room numbers would of course take additional time.
All other steps were performed on-site using our smartphone app for localization, which can be seen in fig. \ref{fig:yasmin}. 
As the museum did not provide any Wi-Fi infrastructure, we installed the \SI{42}{} beacons as explained above. 
Thanks to the great support of the museum's janitor, this step took only \SI{30}{\minute}, as he was well aware of all available power outlets and also helped plugging them in.
After that, each of the \SI{133}{} reference points was scanned 30 times ($\approx \SI{25}{\second}$ scan time) using a Motorola Nexus 6 at \SI{2.4}{GHz}. 
This took \SI{85}{\minute}, as all measurements were conducted using the same smartphone.
The optimized Wi-Fi model and the mesh can be created automatically within a few seconds directly on the smartphone, which then enables the pedestrian to start the localization. 
Of course, for the experiments conducted below several additional knowledge was obtained to evaluate the quality of the proposed methods and the overall localization error. 
Thus the above provided times were measured for a pure localization installation, as for example a customer would order, while the experiments were performed in a 2-day period.
Nevertheless, we believe that an on-site setup-time of less then \SI{120}{\minute} is a big step for the practicability of localization systems. 
In addition, the above steps do not require a high level of detail in their execution, which should also allow unbiased persons to set up the system.}

\begin{figure}[t]
	\centering
	\begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{gfx/apps/yasmin.png}
        \caption{Localization App}
 		\label{fig:yasmin}
    \end{subfigure}
	\hspace{1cm}
	\begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{gfx/apps/simple.png}
        \caption{Simple Recording App}
		 \label{fig:simple}
    \end{subfigure}

    \caption{\add{The two mobile applications developed for Android. The localization app in (a) is used to record the Wi-Fi reference measurements based on the positions provided by the floor plan. In this screenshot the dialog for recording them is visible. The app also implements the here presented approach and can thus be used for localization. However, for the utilized experiments we used a simpler client (b) allowing for user input like a ground truth or activity button.}}
    \label{fig:applications}
\end{figure}

\add{As mentioned, the here presented localization system was implemented as an Android App.
It was written in high performant C++ code, enabling to run completely on the smartphone and thus not requiring any connection to a server.
However, since the experiments required a lot of different information to evaluate the methods, a second, very simple application was developed to record them (see fig. \ref{fig:simple}).
It implements the standard Android sensor functionalities and provides a very simple user interface so that even non-technical users can use it.} 
As smartphones we used either a Samsung Note 2, Google Pixel One or Motorola Nexus 6.
The computation of the state estimation as well as the \docWIFI{} optimization are done offline using an Intel Core i7-4702HQ CPU with a frequency of \SI{2.2}{GHz} running \add{\SI{8}{threads} on \SI{4}{cores}} and \SI{16}{GB} main memory. 
\add{An offline computation has practical advantages, such as easier evaluation of the results or shorter waiting times due to higher computing power. 
Nevertheless, Android app and offline application are both based on the same C++ backend for localization.}


%However, similar to our \add{previously presented system}, the setup is able to run completely on commercial smartphones as it \add{is} written in high performant C++ code \cite{torres2017smartphone}. 
%Sensor measurements are recorded using a simple mobile application that implements the standard Android SensorManager. 

The experiments are separated into five sections:
At first, we discuss the performance of the novel transition model and compare it to a grid-based approach.  
In section \ref{sec:exp:opti} we have a look at \docWIFI{} optimization and how the real \docAPshort{} positions differ from it. 
Following, we conducted several test walks throughout the building to examine the estimation accuracy (in \SI{}{\meter}) of the localization system and discuss the here presented solutions for sample impoverishment.
\add{In section \ref{sec:eval:act} the threshold-based activity recognition is evaluated, providing a detection rate for the test walks utilized before.}
Finally, the respective estimation methods are discussed in section \ref{sec:eval:est}.

\subsection{Transition}

\begin{figure}[t]
	\centering
	\begin{subfigure}{0.4\textwidth}
		\def\svgwidth{\columnwidth}
		\input{gfx/transEval/mesh_25_final.eps_tex}
        \caption{Mesh after 25 steps}
		 \label{fig:transitionEval:a}
    \end{subfigure}
	\hspace{2cm}
	\begin{subfigure}{0.4\textwidth}
		\def\svgwidth{\columnwidth}
		\input{gfx/transEval/grid_25_final.eps_tex}
        \caption{Graph after 25 steps}
 		\label{fig:transitionEval:b}
    \end{subfigure}
	\begin{subfigure}{0.4\textwidth}
		\def\svgwidth{\columnwidth}
		\input{gfx/transEval/mesh_180_final.eps_tex}
        \caption{Mesh after 180 steps}
		 \label{fig:transitionEval:c}
    \end{subfigure}
	\hspace{2cm}
	\begin{subfigure}{0.4\textwidth}
		\def\svgwidth{\columnwidth}
		\input{gfx/transEval/grid_180_final.eps_tex}
        \caption{Graph after 180 steps}
 		\label{fig:transitionEval:d}
\end{subfigure}

    \caption{Simple staircase scenario to compare the \add{old} graph-based model with the \add{new} navigation mesh. All units are given in meter. The black line indicates the current position and the green line gives the estimated path until 25 or 180 steps, both using weighted average. The particles are colored according to their \add{$z$-coordinate}. A pedestrian walks up and down the stairs several times in a row. After 25 steps, both methods produce good results, although there are already some outliers (blue particles). After 180 steps, the outliers using the graph have multiplied, leading to a multimodal situation. In contrast, the mesh offers the possibility to remove particles that hit a wall and can thus prevent such a situation.}
    \label{fig:transitionEval}
\end{figure}

To compare our old graph-based model with our novel transition model presented within section \ref {sec:transition}, we chose a simple scenario, in which a tester walks up and down a staircase several times. 
We used \SI{5000}{} particles and did not perform an evaluation and resampling step to maintain the pure performance of the transition (step and heading).
\add{The number of particles was heuristically chosen and is based on our previous experience from other scenarios and competitions.
In addition, it sill allows a stable performance of our Android app for localization.}
The filter starts at a fixed position and is updated after every newly recognized step. 
We set $\sigma_\text{step} = 0.1$ and $\sigma_\text{turn} = 0.1$ likewise. 
The cells of the gridded graph were \SI{20}{} x \SI{20}{\centi\meter} in size and the transition implemented as described in \cite{Ebner-16}. 
As discussed in section \ref {sec:transition}, the mesh demands for a strategy, how to handle unreachable destinations. 
We chose a simple, yet effective strategy: whenever a destination is unreachable to a particle, it is removed and the last correct transitioning particle is duplicated.
Of course, the graph does not require for such a rule, since particles are only allowed to move on nodes and search for neighbours. 

Fig. \ref{fig:transitionEval:a} and \ref{fig:transitionEval:b} illustrate the results after \SI{25}{steps} for each method. 
The particles are colored according to their height and the walking path (green line) is estimated using weighted-average.
It can be seen that both methods provide similar results.
Due to the discrete grid structure, the purple particles on the graph scatter more strongly, while the mesh provides a truly continuous structure and thus a more compact representation.
It is important to note that outliers have already appeared in both scenarios (blue particles). 
Due to the included sensor noise, they covered a too short distance for several times and thus the upcoming left turn leads upwards instead of downwards.
Going straightforward to \SI{180} steps, this phenomenon has multiplied for the graph (cf. fig. \ref{fig:transitionEval:d}), but not for the mesh (cf. fig. \ref{fig:transitionEval:c}). 
This is due to the above-mentioned strategy for the mesh.
Compared to this approach, the graph is not able to remove any particles and thus they walk according to the recognized steps and heading changes, even if they theoretically hit a wall several times.
The resulting effects are obvious. 
After walking up and down twice, several particle groups have formed, which no longer allows an accurate position estimation.

Of course, a similar strategy could be developed for a graph. 
We have already shown how to identify the nodes nearest to walls in one of our previous works \cite{Ebner-16}. 
However, the limitation to walk in \SI{45}{\degree} angles as well as the discrete cell sizes lead to restrictions for small rooms, narrow hallways or bigger cells. 
For example walking through a door, would result in a strong reduction of differing particles.
If the state evaluation is then used to assign weights to particles, the crucial problem of sample degeneracy often occurs. 
With a mesh, on the other hand, walkable destinations can also be located in a room behind a wall.
In combination with the continues movement, this allows for a high versatility of particles even in such situations. 
Another method to fix the problems shown in fig. \ref{fig:transitionEval:d}, is by adding an activity recognition (walking up, down straight) or to incorporate a barometer. 
Nevertheless, in most cases it is an advantage if a sensor model delivers good results on its own, without further dependencies. 
For example, if a sensor is currently unavailable or damaged, the system is still able to provide prober results. 

Besides the advantages the mesh offers, it also has a few disadvantages compared to the graph.
The computation time has increased due to the calculation of reachable destinations. 
With the graph, only the direct neighbours are of interest, which can be implemented very efficiently using a tree structure.
Further, the graph allows the easily store meta information on its nodes, for example Wi-Fi fingerprints or precalculations for shortest-path methods.
This is more difficult using the mesh and requires the handling of baricentric coordinates.


\subsection{\docWIFI{} Optimization}
\label{sec:exp:opti}

%\commentByToni{Work in Progress... Irgendwie passt die Grafik nicht so wirklich. Im Gegensatz zum 2017 Paper würde ich gerne ein wenig über die geschätzten Positionen reden. Die Unterschiede zwischen Local und Global dabei. Warum machne Schätzungen gar so weit weg von der Realität sind und das es oft auch gar nicht so schlimm ist, falls das passiert. Tipps sind Willkommen. Vielleicht b) weglassen und in a einfach noch die fingerprint positionen mit rein. damit man ein gefühlt dafür bekommt wie viel wir in Vorleistung gehen müssen. An sich erkannt man ja dann das von "oben" das die optimierung manchmal gut und manchmal schlecht ist.}

%wie viele ap sind es insgesamt?
%As described in section \ref{sec:wifi} we used \SI{42}{} WEMOS D1 mini to provide a \docWIFI{} infrastructure throughout the building.
Within all Wi-Fi observations, we only consider the beacons, which are identified by their well-known MAC address. 
Other transmitters like smart TVs or smartphone hotspots are ignored as they might cause estimation errors.
The references (fingerprints) we used to optimize the Wi-Fi model as well as the real position of the \docAPshort{}s (black dot) can be seen in fig. \ref{fig:apfingerprint} for ground level.
\add{For the complete building we defined \SI{133}{} reference points for recording Wi-Fi scans. 
An evaluation of how the number of references affects the optimization can be found in \cite{Ebner-17}.
Increasing their number improves the result only up to a certain factor, which is why we have made a reasonable compromise between recording time and accuracy by distributing them every \SI{3}{\meter} to \SI{7}{\meter} from each other.}
Each reference location was then scanned \SI{30}{} times ($\approx \SI{25}{\second}$ scan time) using a Motorola Nexus 6 at \SI{2.4}{GHz} band only. 
The resulting measurements were grouped per physical transmitter and aggregated to form the average signal strength per transmitter.
The real position of every installed beacon was measured using a laser scanner. 
This allows a comparison with the optimized \docAPshort{} positions, what can also be seen in fig. \ref{fig:apfingerprint}.

%
\begin{figure}[t]
	\centering
	\def\svgwidth{\columnwidth}
	\input{gfx/optimization/wifiOptTopView.eps_tex}
    \caption{Ground level of the building in the \add{$xy$-plane} from above. Includes the locations of the reference points, the ground truth and the optimized \docAPshort{}s. The grey line connects an \docAPshort{} with the corresponding optimization. The \add{colored borders} are areas of special interest and are discussed within the text. The corresponding pictures on the right side show the museum in these places.}
    \label{fig:apfingerprint}
\end{figure}
%Positionsfehler und wo?
It illustrates the results of the global (blue) and the per-floor (orange) method for all \docAPshort{}'s installed to ground level. 
The respective optimized positions $\mPosAPVec$ are connected by a grey line with the corresponding ground truth, providing the position error on the \add{$xy$-plane}.
The average distance error (3D) between the \docAPshort{}'s real position and the optimized ones is \SI{5.4}{\meter} ($\mu =$ \SI{5.1}{}) for the per-floor and \SI{4.8}{\meter} ($\mu =$ \SI{5.6}{}) for global strategy.
However, it is easy to see that the results are better in some areas (green) than in others (red and purple).
While the green \add{rectangle} encloses an area that has a high number of \docAPshort{}s with line-of-sight conditions, the \docAPshort{}s in red and purple are shielded by very thick stone walls and have a lower number of reference points with direct visual contact (cf. fig. \ref{fig:apfingerprint}).
The maximum position error for the global scheme is \SI{25.3}{\meter} and \SI{18.4}{\meter} for the per-floor one.
Both are related to the \docAPshort{} in the red \add{rectangle}.

%Gesamtfehler und wieso?
Of course, the position alone does not provide sufficient information of the overall performance, however it provides a good visual impression of how the optimization behaves.
The overall optimization error is measured using the difference between model predictions and real-world RSSI values for each reference measurement. 
These differences can be positive or negative, which is why we indicate an absolute or signed error.
The (absolute) optimization error of the respective strategies is \SI{4.7}{\decibel} ($\mu =$ \SI{3.8}{}) for global and \SI{2.6}{\decibel} ($\mu =$ \SI{2.7}{}) for per-floor in average.
Again, the highest errors occur from \docAPshort{}s within the red and purple area, whereby the local maxima for the signed difference are \SI{-31.4}{\decibel} and \SI{17.5}{\decibel} for global and \SI{-12.7}{\decibel} and \SI{13.4}{\decibel} for local.
%global vs local
Thus, the per-floor optimization scheme provides a smaller overall error, whereby the positioning error is higher compared to the global one. 
The reason for the latter can be found within the purple area. 
It marks a vaulted cellar, that is \SI{1.7}{\meter} deeper than ground level and connect by a narrow staircase.
\add{Here, RSSI measurements received from \docAPshort{}'s residing on the ground level are strongly attenuated due to the massive walls of the cellar.
In contrast, measurements coming from the floor above are much less attenuated thanks to a much thinner ceiling.}
%Here, RSSI measurements taken from outside the ground level are strongly attenuated, while measurements \del{taken from above are more moderately attenuated} \add{received }. 
Since the per-floor scheme uses only references from the current floor in question, while the global scheme uses all available references and thus more meaningful information in this area. 
However, as the overall error suggests, this is not always an advantage, which we will see later on in the localization experiments.

%warum ist die optimierung tdz. ganz gut?
As mentioned above, some areas are heavily attenuated by big walls, what simply does not fit the used signal strength prediction model. 
As discussed in section \ref{sec:relatedWork} and \ref{sec:wifi}, we only consider ceilings within the model to avoid computational expensive wall intersection-tests. 
A far higher number of reference measurements in bad areas can therefore only increase the accuracy to a limited extent. 
Nevertheless, by optimizing all parameters (\mPosAPVec{}, \mTXP{}, \mPLE{} and \mWAF{}) the system provides far better localization results compared to using the \docAPshort{}'s real positions with empirical values or even optimized values only for \mTXP{}, \mPLE{} and \mWAF{}. 
The reason for this is obvious.
The optimized parameters fit the (unrealistic) signal strength prediction model much better than the real ones and thus provide for a smaller error between measured RSSI and predicted RSSI. 
Since walls are ignored by the model, optimizing the position of the access points can compensate for the resulting effects. 
This is also the reason why the optimized positions of \docAPshort{}'s attached to walls always have a certain distance to them, as can be seen in fig. \ref{fig:apfingerprint}.
A more realistic model would not only mean an overall improvement of the results, but also a further approximation to the real conditions in the building. It is to be expected that the estimated positions of the access points will then approach the ground truth. 
Further evaluations and discussions regarding the here used optimization can be found in \cite{Ebner-17}.


\subsection{Localization Error}
\label{sec:exp:loc}

\begin{figure}[t]
	\centering
	\begin{subfigure}{0.32\textwidth}
		\def\svgwidth{\columnwidth}
		{\input{gfx/groundTruth/gt_unten_final.eps_tex}}
        \caption{Ground floor}
    \end{subfigure}
	\begin{subfigure}{0.32\textwidth}
		\def\svgwidth{\columnwidth}
		{\input{gfx/groundTruth/gt_mitte_final.eps_tex}}
        \caption{First floor}
    \end{subfigure}
	\begin{subfigure}{0.32\textwidth}
		\def\svgwidth{\columnwidth}
		{\input{gfx/groundTruth/gt_oben_final.eps_tex}}
        \caption{Second floor}
    \end{subfigure}
	\caption{All conducted walks within the building. The arrows indicate the running direction and a cross marks the end. For a better overview we have divided the building into three floors, \add{which are connected by four stairs (numbered 1--4)}. However, each floor consists of different high levels. They are separated from each other by different shades of grey, dark is lower than light.}
	\label{fig:floorplan}
\end{figure}
%
The 4 chosen walking paths can be seen in fig. \ref{fig:floorplan}. 
Walk 0 is \SI{152}{\meter} long and took about \SI{2.30}{\minute} to walk.
Walk 1 has a length of \SI{223}{\meter} and Walk 2 a length of \SI{231}{\meter}, both required about \SI{6}{\minute} to walk. 
Finally, walk 3 is \SI{310}{\meter} long and takes \SI{10}{\minute} to walk.
All walks were carried out by 4 different male testers using either a Samsung Note 2, Google Pixel One or Motorola Nexus 6 for recording the measurements. 
All in all, we recorded \SI{28}{} distinct measurement series, \SI{7}{} for each walk.
The picked walks intentionally contain erroneous situations, in which many of the above treated problems occur. 
\del{This allows us to discuss everything in detail.}
A walk is indicated by a set of numbered markers, fixed to the ground. 
Small icons on those markers give the direction of the next marker and in some cases provide instructions to pause walking for a certain time.
The intervals for pausing vary between \SI{10}{\second} to \SI{60}{\second}. 
The ground truth is then measured by recording a timestamp while passing a marker. 
For this, the tester clicks a button on the smartphone application. 
Between two consecutive points, a constant movement speed is assumed. 
Thus, the ground truth might not be \SI{100}{\percent} accurate, but fair enough for error measurements. 
The approximation error is then calculated by comparing the interpolated ground truth position with the current estimation \cite{Fetzer-16}.
An estimation on the wrong floor has a great impact on the location awareness of an pedestrian, but only provides a relatively small error.
Therefore, errors in $z$-direction are penalized by tripling the $z$-value.

%computation und monte carlo runs
For each walk we deployed 100 runs using \SI{5000}{particles} and set $N_{\text{eff}} = 0.85$ for resampling. 
Instead of an initial position and heading, all walks start with a uniform distribution (random position and heading) as prior.
The overall localization results can be see in table \ref{table:overall}. 
Here, we differ between the respective anti-impoverishment techniques presented in chapter \ref{sec:impo}.
The simple anti-impoverishment method is added to the resampling step and thus uses the transition method presented in chapter \ref{sec:transition}.
In contrast, the $D_\text{KL}$-based method extends the transition and thus uses a standard cumulative resampling step. 
We set $l_\text{max} =$ \SI{-75}{dBm} and $l_\text{min} =$ \SI{-90}{dBm}.
For a better overview, we only used the KDE-based estimation, as the errors compared to the weighted-average estimation differ by only a few centimeter. 

\begin{table}[t]
	\centering
	\begin{tabular}{rrrrcrrrcrrr}
		\toprule
		 & \multicolumn{3}{c}{none} &  \phantom{abc} & \multicolumn{3}{c}{simple} & \phantom{abc} &  \multicolumn{3}{c}{$D_\text{KL}$} \\
		 \cmidrule{2-4}  \cmidrule{6-8} \cmidrule{10-12}
		 & \multicolumn{1}{c}{$\bar{x}$} & \multicolumn{1}{c}{$\bar{\sigma}$} & \multicolumn{1}{c}{$\tilde{x}_{75}$} && \multicolumn{1}{c}{$\bar{x}$} & \multicolumn{1}{c}{$\bar{\sigma}$} & \multicolumn{1}{c}{$\tilde{x}_{75}$} && \multicolumn{1}{c}{$\bar{x}$} & \multicolumn{1}{c}{$\bar{\sigma}$} & \multicolumn{1}{c}{$\tilde{x}_{75}$} \\ 
		\midrule 
     	walk 0 & \SI{13.4}{\meter} & \SI{11.2}{\meter} & \SI{22.6}{\meter} && \SI{7.1}{\meter} & \SI{6.6}{\meter} & \SI{9.4}{\meter} && \SI{5.8}{\meter} & \SI{4.9}{\meter} & \SI{7.3}{\meter} \\ 
		walk 1 & \SI{3.2}{\meter}  & \SI{2.4}{\meter}  & \SI{4.1}{\meter}  && \SI{3.2}{\meter} & \SI{2.6}{\meter} & \SI{4.0}{\meter} && \SI{3.8}{\meter} & \SI{3.2}{\meter} & \SI{4.6}{\meter} \\ 
		walk 2 & \SI{8.3}{\meter}  & \SI{4.1}{\meter}  & \SI{10.9}{\meter} && \SI{3.6}{\meter} & \SI{2.3}{\meter} & \SI{4.9}{\meter} && \SI{3.6}{\meter} & \SI{2.3}{\meter} & \SI{4.8}{\meter} \\ 
		walk 3 & \SI{7.0}{\meter}  & \SI{5.9}{\meter}  & \SI{13.5}{\meter} && \SI{5.4}{\meter} & \SI{4.7}{\meter} & \SI{7.7}{\meter} && \SI{4.8}{\meter} & \SI{4.3}{\meter} & \SI{6.5}{\meter} \\
		\bottomrule
	\end{tabular}
		\caption{Overall localization results in meter using the different impoverishment methods. For estimation we used the KDE-based method, as the errors compared to the weighted-average differ by only a few centimeter. The results are presented given the average positioning error $\bar{x}$, the standard deviation $\bar{\sigma}$ and the \SI{75}{\percent}-quantil of positioning errors $\tilde{x}_{75}$.}
	\label{table:overall}
\end{table}
All walks, except for walk 1, suffer in some way from sample impoverishment. 
We discuss the single results of table \ref{table:overall} starting with walk 0. 
Here, the pedestrians started at the top most level, walking down to the lowest point of the building. 
The first critical situation occurs immediately after the start.
While walking down the small staircase, many particles are getting dragged into the room to the right due to erroneous Wi-Fi readings. 
At this point, the activity "walking down" is recognized, however only for a very short period. 
This is caused by the short length of the stairs.
After this period, only a small number of particles changed the floor correctly, while a majority is stuck within the right-hand room.
The activity based evaluation $p(\vec{o}_t \mid \vec{q}_t)_\text{act}$ prevents particles from further walking down the stairs, while the resampling step mainly draws particles in already populated areas.
In \SI{10}{\percent} of the runs using none of the anti-impoverishment methods, the system is unable to recover and thus unable to finish the walk somewhere near the correct position or even on the same floor. 
Yet, the other \SI{90}{\percent} of runs suffer from a very high error. 
Only by using one of the here presented methods to prevent impoverishment, the system is able to recover in \SI{100}{\percent} of cases.
Fig. \ref{fig:errorOverTimeWalk0} compares the error over time between the different methods for an exemplary run.
The above described situation, causing the system to stuck after \SI{10}{\second}, is clearly visible.  
Both, the simple and the $D_\text{KL}$ method are able to recover early and thus decrease the overall error dramatically.
Between \SI{65}{\second} and \SI{74}{\second} the simple method produces high errors due to some uncertain Wi-Fi measurements coming from an \docAP{} below, causing those particles who are randomly drawn near this \docAPshort{} to be rewarded with a very high weight. 
This leads to newly sampled particles in this area and therefore a jump of the estimation. 
The situation is resolved after entering another room, which is now shielded by stone walls instead of wooden ones. 
Walking down the stairs at \SI{80}{\second} does also recover the localization system using none of the methods.
%
\begin{figure}
	\centering
	\input{gfx/errorOverTimeWalk0/errorOverTime.tex}
	\caption{Error development over time of a single particle filter run of walk 0. Between \SI{10}{\second} and \SI{24}{\second} the Wi-Fi signal was highly attenuated, causing the system to get stuck and producing high errors. Both, the simple and the $D_\text{KL}$ anti-impoverishment method are able to recover early. However, between \SI{65}{\second} and \SI{74}{\second} the simple method produces high errors due to the high random factor involved.} 
	\label{fig:errorOverTimeWalk0}
\end{figure}

A similar behavior as the above can be seen in walk 3. 
Without a method to recover from impoverishment, the system lost track in \SI{100}{\percent} of the runs due to a not detected floor change in the last third of the walk. 
By using the simple method, the overall error can be reduced and the impoverishment resolved. Nevertheless, unpredictable jumps of the estimation are causing the system to be highly uncertain in some situations, even if those jumps do not last to long.
Only the use of the $D_\text{KL}$ method is able to produce reasonable results.

As described in chapter \ref{sec:wifi}, we use a Wi-Fi model optimized for each floor instead of a single global one.
A good example why we do this, can be seen in fig. \ref{fig:walk3:wifiopt}, considering a small section of walk 3.
Here, the system using the global Wi-Fi model makes a big jump into the right-hand corridor and requires \SI{5}{\second} to recover.
This happens through a combination of environmental occurrences, like the many different materials and thus attenuation factors, as well as the limitation of the here used Wi-Fi model, only considering ceilings and ignoring walls.
Following, \docAPshort{}'s on the same floor level, which are highly attenuated by \SI{2}{\meter} thick stone walls, are neglected and \docAPshort{}'s from the floor above, which are only separated by a thin wooden ceiling, have a greater influence within the state evaluation process.
Of course, we optimize the attenuation per floor, but at the end this is just an average value summing up the \docAPshort{}'s surrounding materials. 
Therefore, the calculated signal strength predictions do not fit the measurements received from the above in a optimal way.
In contrast, the model optimized for each floor only considers the respective \docAPshort{}'s on that floor, allowing to calculate better fitting parameters. 
A major disadvantage of the method is the reduced number of visible \docAPshort{}'s and thus measurements within an area. 
This could lead to an underrepresentation of \docAPshort{}'s for triangulation. 
Such a scenario can be seen in fig. \ref{fig:walk3:time} between \SI{200}{\second} and \SI{220}{\second}, where the pedestrian enters an isolated room. 
Only two \docAPshort{}'s provide a solid signal within this area, leading to a higher error, while the global scheme still receives RSSI readings from above.

\begin{figure}[t!]
	\centering
  	\begin{subfigure}[t]{0.45\textwidth}
		\def\svgwidth{\columnwidth}
	    {\input{gfx/wifiOptGlobalFloor/wifiOptGlobalFloor.eps_tex}}
        \caption{}
        \label{fig:walk3:wifiopt}
    \end{subfigure}
    \hfil
	\begin{subfigure}[t]{0.45\textwidth}
		\resizebox{1\textwidth}{!}{\input{gfx/errorOverTimeWalk3/errorOverTime.tex}}
        \caption{}
        \label{fig:walk3:time}
    \end{subfigure}
	\caption{(a) A small section of walk 3. Optimizing the system with a global Wi-Fi optimization scheme (blue) causes a big jump and thus high errors. This happens due to highly attenuated Wi-Fi signals and inappropriate Wi-Fi parameters. We compare this to a system optimized for each floor individually (orange), resolving the situation a producing reasonable results. (b) Error development over time for this section. The high error can be seen at \SI{190}{\second}. }
	\label{fig:walk3}
\end{figure}


%walk 1 
Looking at the results of table \ref{table:overall} again, it can be seen that the $D_\text{KL}$ method is able to improve the results in three of the four walks. 
Those walks have in common, that they suffer in some way from sample impoverishment or other problems causing the system to stuck. 
The only exception is walk 1. 
It was set up to provide a challenging scenario, leading to as many multimodalities as possible.
We intentionally searched for situations in which there was a great chance that the particle set would separate, e.g. by providing multiple possible whereabouts through crossings or by blocking and thus separating a straight path with objects like movable walls. 
Similar to the other walks, we added different pausing intervals of \SI{10}{\second} to \SI{60}{\second}. 
This helps to analyse how the particles behave in such situations, especially in this multimodal setting.

Besides uncertain measurements, one of the main sources for multimodalities are restrictive transition models, e.g. no walking through walls. 
As shown in section \ref{sec:impo}, the $D_\text{KL}$ method compares the current posterior $p(\mStateVec_{t} \mid \mObsVec_{1:t})$ with the probability grid $\probGrid_{t, \text{wifi}}$ using the Kullback-Leibler divergence and a Wi-Fi quality factor. 
Environmental restriction like walls are not considered while creating $\probGrid_{t, \text{wifi}}$, that is why the grid is not effected by a transition-based multimodal setting. 
Given accurate Wi-Fi measurements, it is therefore very likely that $\probGrid_{t, \text{wifi}}$ represents a unimodal distribution, even if the particles got separated by an obstacle or wall. 
This leads to a situation, in which posterior and grid differ.
As a result, the radius $r_\text{sub}$ increases and thus the diversity of particles. 
We are able to confirm the above by examining the different scenarios integrated into walk 1.
For this, we compared the error development with the corresponding radius $r_\text{sub}$ over time. 
In situations where the errors given by the $D_\text{KL}$ method and the simple method differ the most, $r_\text{sub}$ also increases the most. 
Here, the radius grows to a maximum of $r_\text{sub} = $ \SI{8.4}{\meter}, using the same measurement series as in fig. \ref{fig:walk1:kdeovertime}.
In contrast, a real sample impoverishment scenario, as seen in walk 0 (cf. fig. \ref{fig:errorOverTimeWalk0}), has a maximum radius of \SI{19.6}{\meter}.
Nevertheless, such an slightly increased diversity of \SI{8.4}{\meter} is enough to influence the estimation error of the $D_\text{KL}$ in a negative way (cf. walk 1 in table \ref{table:overall}). 
Ironically, this is again some type of sample impoverishment, caused by the aforementioned environmental restrictions not allowing particles inside walls or other out of reach areas. 


%%activity
\subsection{Activity Recognition}
\label{sec:eval:act}

\add{In order to evaluate the activity recognition, a test person had to press a button according to their current state of motion, namely standing, walking, walking up, walking down, elevator up and elevator down (cf. fig. \ref{fig:simple}). 
As the building does not have an elevator, this state is ignored in the following. 
Whether a state needs to be changed was indicated by small symbols on the ground truth markers. 
This experiment is based on the same measurement series as section \ref{sec:exp:loc}.}
%
\add{As the activity recognition uses moving averages, the detection suffers from a certain lag, depending on their size. 
Thus, comparing each activity that is newly calculated with incoming barometer measurements to the ground truth at the current timestamp, would result in a rather low detection rate for the respective activities.
In addition, only a fraction of a test path consists of the change of an activity, since the testers were walking most of the time.  
This would bias an overall detection rate.}
%It was also investigated, that the standing activity could hardly be recognized, because the test persons constantly moved and turned around to look at the exhibits. 
%As a result, a proper evaluation of this activity could not be carried out, so we only evaluate the floor changes. 
%Grafik die das zeigt. 

\begin{table}[t]
	\centering
	\begin{tabular}{cccccc}
		\toprule
		 & standing & walking & walking up & walking down & overall \\ 
		\midrule 
 		walk 0 & \SI{65.6}{\percent} 	& \SI{80.9}{\percent} 	& - 					& \SI{84.8}{\percent} 	& \SI{78.4}{\percent} \\ 
		walk 1 & \SI{49.9}{\percent}  	& \SI{84.1}{\percent}  	& -  					& - 					& \SI{67.5}{\percent} \\ 
		walk 2 & \SI{57.4}{\percent}  	& \SI{83.5}{\percent}  	& \SI{83.5}{\percent} 	& \SI{82.1}{\percent} 	& \SI{71.7}{\percent} \\ 
		walk 3 & \SI{45.7}{\percent}  	& \SI{77.5}{\percent}  	& \SI{85.1}{\percent} 	& \SI{77.8}{\percent} 	& \SI{61.3}{\percent} \\
		\midrule 		
		overall & \SI{51.4}{\percent}  & \SI{81.5}{\percent}  & \SI{84.3}{\percent} & \SI{82.1}{\percent} & \SI{67.9}{\percent} \\
		\bottomrule
	\end{tabular}
		\caption{\add{The resulting detection rates provided by the activity recognition for all conducted walks. As the method suffers from a (time) lag, caused by the used moving average, we shifted the measured activity according to the average lag over all walks (\SI{2.96}{\second}). Some cells of the table are empty, because the respective walk did not require this activity.}}
	\label{table:activity}
\end{table}

\add{In order to be able to make a statement about the quality, we first determined the average (time) lag of the conducted walks and then shifted the calculated data accordingly.
This does not allow a perfect, but at least fair examination of the detection rate.
The lag is given as the (absolute) difference between the timestamp, the activity swaps in ground truth (e.g. from standing to walking), and the first timestamp of an interval, given by the size of $\vec{\omega}_\text{s}$, holding the same activity, given by our recognition method.  
This provides in an average lag of \SI{2.96}{\second} and a standard deviation of \SI{1.09}{\second} over all walks. 
The resulting detection rates can be seen in table \ref{table:activity}.
They are calculated by dividing the number of correctly detected activities with the number of activities given by the ground truth.
The first thing to notice is the bad recognition rate of standing, especially in comparison to the others.
A major impact on this, is the fact, that we encourage the testers to behave as natural as possible, i.e. like a normal visitor of the museum. 
As a result, they often turned around or a took a few small steps within the standing sequences, to look at the exhibits.
This behavior is not mapped by the ground truth. 
In addition, using only acceleration for detecting might be a bad choice in the first place, as moving the phone, e.g. by putting it in the trouser pocket, will exceed the threshold.  
At the end, this leads to the general question, on how to define standing. 
Is it a complete standstill or should it allow for a certain degree of freedom? 
The answer is always the same, it depends. 
As for this museum scenario, the results for detecting the standing activity are not satisfying and a more advanced approach should be considered.}


\add{The detection rates of up and down sind nicht ganz so schlecht, as the main reason ... 
all these changes could be detect with} 


Finally, it is not possible to derive the smartphone or the user from the recognition rates, which recommends a general use of the approach. 

%das brauchen wir glaub nimmer. das oben ist schon nicht verkehrt. 
%\add{The main reason to utilize such a method was to detect floor changes.
%Regardless of the detection rate above, the method is able to detect all floor changes of the conducted walks.
%This was quantified by comparing ob in JEDEM ground truth interval min. 75 prozent korrekt erkannte aktivitäten sind. }



%%estimation
\subsection{Estimation}
\label{sec:eval:est}

As mentioned before, the single estimation methods (cf. chapter \ref{sec:estimation}) only vary by a few centimetres in the overall localization error. 
That means, they differ mainly in the representation of the estimated locations. 
More easily spoken, in which way the estimated path is drawn and thus presented to the user.
Regarding the underlying particle set, different shapes of probability distributions need to be considered, especially those with multimodalities. 
%
\begin{figure}[t]
	\centering
	\begin{subfigure}{0.45\textwidth}
		\resizebox{1\textwidth}{!}{\input{gfx/walk.tex}}
        \caption{}
        \label{fig:walk1:kde}
    \end{subfigure}
    \hfil
	\begin{subfigure}{0.45\textwidth}
		\resizebox{1\textwidth}{!}{\input{gfx/errorOverTimeWalk1/errorOverTime.tex}}
        \caption{}
        \label{fig:walk1:kdeovertime}
    \end{subfigure}
	\caption{(a) Occurring bimodal distribution caused by uncertain measurements in the first \SI{13.4}{\second} of walk 1. After \SI{20.8}{\second}, the distribution gets unimodal. The weigted-average estimation (orange) provides a high error compared to the ground truth (solid black), while the KDE approach (blue) does not. (b) Error development over time for the complete walk. From \SI{230}{\second} to \SI{290}{\second} to pedestrian was not moving. }
	\label{fig:walk1}
\end{figure}
%
The main advantage of a KDE-based estimation is that it provides the "correct" mode of a density, even under a multimodal setting (cf. section \ref{sec:estimation}). 
That is why we again have a look at walk 1.
A situation in which the system highly benefits from this is illustrated in fig. \ref{fig:walk1:kde}.
Here, a set of particles splits apart, due to uncertain measurements and multiple possible walking directions.
Indicated by the black dotted line, the resulting bimodal posterior reaches its maximum distance between the modes at \SI{13.4}{\second}.
Thus, a weighted-average estimation (orange line) results in a position of the pedestrian somewhere outside the building (light green area). 
The ground truth is given by the black solid line.
The KDE-based estimation (blue line) is able to provide reasonable results by choosing the "correct" mode of the density.
After \SI{20.8}{\second} the setting returns to be unimodal again. 
Due to a right turn the lower red particles are walking against a wall and thus punished with a low weight.

Although, situations as displayed in fig. \ref{fig:walk1:kde} frequently occur, the KDE-estimation is not able to improve the overall estimation results.
This can be seen in the corresponding error development over time plot given by fig. \ref{fig:walk1:kdeovertime}. 
Here, the KDE-estimation performs slightly better then the weighted-average, however after deploying \SI{100}{} runs of the particle filter, the difference becomes insignificant.
It is obvious, that the above mentioned "correct" mode, not always provides the lowest error. 
In some situations the weighted-average estimation is often closer to the ground truth. 
Within our experiments this happened especially when entering or leaving thick-walled rooms, causing slow and attenuated Wi-Fi signals. 
While the system’s dynamics are moving the particles outside, the faulty Wi-Fi readings are holding back a majority by assigning corresponding weights.
Only with new measurements coming from the hallway or other parts of the building, the distribution and thus the KDE-estimation are able to recover. 

This leads to the conclusion, that a weighted-average approach provides a more smooth representation of the estimated locations and thus a higher robustness.
A comparison between both methods is illustrated in fig. \ref{fig:estimationcomp} using a measuring sequence of walk 2.
We have highlighted some interesting areas with colored \del{squares} \add{rectangles}.
The greatest difference between the respective estimation methods can be seen inside the green rectangle, the gallery wing of the museum.
While the weighted-average (orange) produces a very straight estimated path, the KDE-based method (blue) is much more volatile. 
This can be explained by the many small rooms that pedestrians pass through. 
The doors act like bottlenecks, which is why many particles run against walls and thus are either drawn on a new position within a reachable area (cf. section \ref{sec:estimation}) or walk along the wall towards the door.
This causes a higher uncertainty and diversity of the posterior, what is more likely to be reflected by the KDE method than by the weighted-average.
Additionally, the pedestrian was forced seven times to look at paintings (stop walking) between \SI{10}{\second} and \SI{20}{\second}, just in this small area. 
Nevertheless, even if both estimated paths look very different, they produce similar errors. 

The purple rectangle displays a situation in which a sample impoverishment was successfully resolved. 
Due to a poorly working \docAPshort{}, in the lower corner of the big room the pedestrians passes before walking down the stairs, the majority of particles is dragged into the upper right corner of  that room and unable to walk down. 
By allowing some particles to walk through the wall and thus down the stairs, the impoverishment could be dissolved. 
The KDE-based estimation \add{(blue line)} illustrates this behavior very accurate. 
\add{At first, the pedestrian's position is estimated in the area around the corner of the room, after the impoverishment was recognized, the estimated path is then crossing the wall, enabling the floor change.
However, as could be seen in fig. 7, before the here presented methods are able to resolve sample impoverishment, the error and thus the radius $r_\text{sub}$ increase in time as the system got stuck. 
This can take up to a few seconds, in which the pedestrian has continued walking and is thus ahead of the current position estimation. 
As the first particles are newly drawn into more proper regions, the system starts to recover, still remaining in an uncertain state as the particle set is split apart.  
After a few filter updates, especially resampling steps, the system returns back to a more stable state. 
Both, the time in which the system was uncertain as well as the lag to the real position of the pedestrian lead to problems at the end of the staircase (left corner of the purple rectangle). 
While the pedestrian has already completely descended the stairs, the activity changes from walking down to walking. 
Since the majority of the particles are still on the stairs, they are thus considered to be less likely than particles on a floor. 
As there are only a few particles on the floors, some below and some above, the estimation is calculated somewhere in between, which finally explains the increased error in this area.}

Another situation in which the estimated paths do not provide sufficient results can be seen inside the teal rectangle. 
The room is very isolated from the rest of the building, which is reflected by the fact that only 3 \docAPshort{}'s are detected.
The pedestrians have been asked to cross the room at a quick pace, leading to a higher step rate and therefore update rate of the filter.
The results within this area lead to the assumption, that even if Wi-Fi has a bad coverage, it influences the estimation results the most. 
The PDR based transition alone is able to walk alongside the ground truth in an accurate manner. 
However, this is of course only true if we consider this area individually, without the rest of the walk due to the accumulating bias of the relative sensors involved.
\del{In the end, it is a question of optimal harmony between transition and evaluation.} 
We hope to further improve such situations in future work by enabling the transition step to provide a weight to particles that walk very likely, especially in situation where Wi-Fi provides bad readings.

\begin{figure}[t]
	\centering
		\def\svgwidth{0.8\columnwidth}
		{\input{gfx/estimationPath2/est.eps_tex}}
	\caption{Estimation results of walk 2 using the KDE method (blue) and the weighted-average (orange). While the latter provides a more smooth representation of the estimated locations, the former provides a better idea of the quality of the underlying processes. In order to keep a better overview, the top level of the last floor was hidden. The colored rectangles are used as references within the text.}
	\label{fig:estimationcomp}
\end{figure}

To summarize, the KDE-based approach for estimation is able to resolve multimodalities. 
It does not provide a smooth estimated path, since it depends more on an accurate sensor model than a weighted-average approach, but is suitable as a good indicator about the real performance of a sensor fusion system. 
At the end, in the here shown examples we only searched for a global maxima, even though the KDE approach opens a wide range of other possibilities for finding a best estimate. 
\add{A detailed examination of the runtime performance of the used estimation methods in comparison to the state-of-the-art can be found in \cite{Bullmann-18}.}


