FHWS/IPIN2018
Archived
4
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

added graphics and text to experiments

Browse Source
This commit is contained in:
toni
2018-07-01 11:09:32 +02:00
parent 1e909acb5d
commit d30d8cbe73
10 changed files with 2283 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
tex/chapters/eval.tex
View File

@@ -106,6 +106,7 @@ Nevertheless, our method also allows to distribute beacons in the whole building
\subsection{Activity Recognition}
\label{sec:activity}
To enable continuous floor changes we use a simple activity recognition based on the smartphone's barometer and accelerometer.
The method distinguishes between the following: standing, walking, walking up or walking down.

96
tex/chapters/experiments.tex
View File

@@ -19,11 +19,11 @@ The computation of the state estimation as well as the \docWifi{} optimization a
However, similar to our previous, award-winning system, the setup is able to run completely on commercial smartphones as well as it uses C++ code \cite{torres2017smartphone}.
%Sensor measurements are recorded using a simple mobile application that implements the standard Android SensorManager.
The experiments are separated into three sections:
The experiments are separated into four sections:
At first, we discuss the performance of the novel transition model and compare it to a grid-based approach.
In section \ref{sec:exp:opti} we have a look at \docWIFI{} optimization and how the real \docAPshort{} positions differ from it.
Following, we conducted several test walks throughout the building to examine the estimation accuracy (in \SI{}{\meter}) of the localisation system.
We try to resolve sample impoverishment with the here presented method and compare the different estimation methods as presented in section \ref{sec:estimation}.
Following, we conducted several test walks throughout the building to examine the estimation accuracy (in \SI{}{\meter}) of the localisation system and try to resolve sample impoverishment with the here presented methods.
Finally, the different estimation methods are compared in section \ref{sec:exp:est}.
\subsection{Transition}
@@ -57,7 +57,7 @@ Other transmitters like smart TVs or smartphone hotspots are ignored as they mig
%was kommt bei der optimierung raus. vergleichen mit ground truth. auch den fehler gegenüberstellen.
%man sollte sehen das ohne optimierung gar nichts geht.
\subsection{Location Estimation Error}
\subsection{Localization Error}
\begin{figure}[ht]
\centering
@@ -66,40 +66,33 @@ Other transmitters like smart TVs or smartphone hotspots are ignored as they mig
\label{fig:floorplan}
\end{figure}
%
The 4 chosen walking paths can be seen in fig. \ref{fig:floorplan}.
The 4 chosen walking paths can be seen in fig. \ref{fig:floorplan}.
\todo{wie lang sind die walks meter und zeit?}
They were carried out be 4 different male testers using either a Samsung Note 2, Google Pixel One or Motorola Nexus 6 for recording the measurements.
All in all, we recorded \SI{28}{} distinct measurement series, \SI{7}{} for each walk.
The picked walks contain erroneous situations, in which many of the above treated problems occur.
The picked walks intentionally contain erroneous situations, in which many of the above treated problems occur.
Thus we are able to discuss everything in detail.
A walk is indicated by a set of numbered markers, fixed to the ground.
Small icons on those markers give the direction of the next marker and in some cases provide instructions to pause walking for a certain time.
The intervals for pausing vary between \SI{10}{\second} to \SI{60}{\second}.
The ground truth is then measured by recording a timestamp while passing a marker.
For this, the tester clicks a button on the smartphone application.
Between two consecutive points, a constant movement speed is assumed.
Between two consecutive points, a constant movement speed is assumed.
Thus, the ground truth might not be \SI{100}{\percent} accurate, but fair enough for error measurements.
The approximation error is then calculated by comparing the interpolated ground truth position with the current estimation \cite{Fetzer-16}.
An estimation on the wrong floor has a great impact on the location awareness of an pedestrian, but only provides a relatively small error.
Therefore, errors in $z$-direction are penalized by tripling the $z$-value.
%computation und monte carlo runs
For each walk we deployed 100 runs using \SI{5000}{particles}.
Instead of an initial position and heading, all walks start with a uniform distribution (random position and heading) as prior.
The overall localisation results can be see in table \ref{table:overall}.
Here, we differ between the respective impoverishment techniques presented in chapter \ref{sec:impo}.
Here, we differ between the respective anti-impoverishment techniques presented in chapter \ref{sec:impo}.
The simple anti-impoverishment method is added to the resampling step and thus uses the transition method presented in chapter \ref{sec:transition}.
In contrast, the $D_\text{KL}$-based method extends the transition and thus uses a standard cumulative resampling step.
We set $l_\text{max} =$ \SI{-75}{dBm} and $l_\text{min} =$ \SI{-90}{dBm}.
For a better overview, we only used the KDE-based estimation, as the errors compared to the weighted average estimation differ by only a few centimetres.
a simple filter (weighted average estimation + simple impoverishment solution) and an advanced filter (KDE estimation + $D_\text{KL}$-based impoverishment solution).
It can be seen that...
the results include also all failed walks, to show the benefits of using an anti impoverishment method. in walk 0 at walk 1 20 percent of walks failed ...
of course it would be possible to resets the system at this point, but thats not our anspruch...
we want to show a real worst case scenario!
\todo{providing a penalty for wrong floors. sonst haben wir das problem das der overall error einfach nicht unterschiedlich genug ist. }
\newcommand{\STAB}[1]{\begin{tabular}{@{}c@{}}#1\end{tabular}}
\begin{table}[t]
@@ -110,36 +103,61 @@ we want to show a real worst case scenario!
\hline
& $\bar{x}$ & $\bar{\sigma}$ & $\tilde{x}_{75}$ & $\bar{x}$ & $\bar{\sigma}$ & $\tilde{x}_{75}$ & $\bar{x}$ & $\bar{\sigma}$ & $\tilde{x}_{75}$ \\
\hline \hline
Walk 0 & \SI{1315}{\centi\meter} & \SI{1136}{\centi\meter} & \SI{2266}{\centi\meter} & \SI{1189}{\centi\meter} & \SI{1092}{\centi\meter} & \SI{2019}{\centi\meter} & \SI{301}{\centi\meter} & \SI{252}{\centi\meter} & \SI{376}{\centi\meter} \\ \hline
Walk 1 & \SI{318}{\centi\meter} & \SI{243}{\centi\meter} & \SI{402}{\centi\meter} & \SI{318}{\centi\meter} & \SI{240}{\centi\meter} & \SI{403}{\centi\meter} & \SI{342}{\centi\meter} & \SI{256}{\centi\meter} & \SI{440}{\centi\meter} \\ \hline
Walk 2 & \SI{589}{\centi\meter} & \SI{403}{\centi\meter} & \SI{843}{\centi\meter} & \SI{321}{\centi\meter} & \SI{210}{\centi\meter} & \SI{431}{\centi\meter} & \SI{342}{\centi\meter} & \SI{219}{\centi\meter} & \SI{455}{\centi\meter} \\ \hline
Walk 3 & \SI{462}{\centi\meter} & \SI{337}{\centi\meter} & \SI{701}{\centi\meter} & \SI{407}{\centi\meter} & \SI{306}{\centi\meter} & \SI{599}{\centi\meter} & \SI{341}{\centi\meter} & \SI{253}{\centi\meter} & \SI{462}{\centi\meter} \\
Walk 0 & \SI{1340}{\centi\meter} & \SI{1115}{\centi\meter} & \SI{2265}{\centi\meter} & \SI{715}{\centi\meter} & \SI{660}{\centi\meter} & \SI{939}{\centi\meter} & \SI{576}{\centi\meter} & \SI{494}{\centi\meter} & \SI{734}{\centi\meter} \\ \hline
Walk 1 & \SI{320}{\centi\meter} & \SI{242}{\centi\meter} & \SI{406}{\centi\meter} & \SI{322}{\centi\meter} & \SI{258}{\centi\meter} & \SI{404}{\centi\meter} & \SI{379}{\centi\meter} & \SI{317}{\centi\meter} & \SI{463}{\centi\meter} \\ \hline
Walk 2 & \SI{834}{\centi\meter} & \SI{412}{\centi\meter} & \SI{1092}{\centi\meter} & \SI{356}{\centi\meter} & \SI{232}{\centi\meter} & \SI{486}{\centi\meter} & \SI{362}{\centi\meter} & \SI{234}{\centi\meter} & \SI{484}{\centi\meter} \\ \hline
Walk 3 & \SI{704}{\centi\meter} & \SI{589}{\centi\meter} & \SI{1350}{\centi\meter} & \SI{538}{\centi\meter} & \SI{469}{\centi\meter} & \SI{782}{\centi\meter} & \SI{476}{\centi\meter} & \SI{431}{\centi\meter} & \SI{648}{\centi\meter} \\
\hline
\end{tabular}
\caption{Overall localization results using the different impoverishment methods. The error is given by the \SI{75}{\percent}-quantil Used only kde for estimation, since kde and avg nehmen sich nicht viel. fehler kleiner als 10 cm im durchschnitt deshalb der übersichtshalber weggelassen. }
\label{table:overall}
\end{table}
All walks, except for walk 1, suffer in some way from sample impoverishment.
We discuss the single results of table \ref{table:overall} starting with walk 0.
Here, the pedestrians started at the top most level, walking down to the lowest point of the building.
The first critical situation occurs immediately after the start.
While walking down the small staircase, many particles are getting dragged into the room to the right due to erroneous Wi-Fi readings.
At this point, the activity "walking down" is recognized, however only a for very short period.
This is caused by the short length of the stairs.
After this period, only a small number of particles changed the floor correctly, while a majority is stuck within the right-hand room.
The activity based evaluation $p(\vec{o}_t \mid \vec{q}_t)_\text{act}$ prevents particles from further walking down the stairs, while the resampling step mainly draws particles in already populated areas.
In \SI{10}{\percent} of the runs using none of the anti-impoverishment methods, the system is unable to recover and thus unable to finish the walk somewhere near the correct position or even on the same floor.
Yet, the other \SI{90}{\percent} of runs suffer from a very high error.
Only by using one of the here presented methods to prevent impoverishment, the system is able to recover in \SI{100}{\percent} of cases.
Fig. \ref{fig:errorOverTimeWalk0} compares the error over time between the different methods for an exemplary run.
The above described situation, causing the system to stuck after \SI{10}{\second}, is clearly visible.
Both, the simple and the $D_\text{KL}$ method are able to recover early and thus decrease the overall error dramatically.
Between \SI{65}{\second} and \SI{74}{\second} the simple method produces high errors due to some uncertain Wi-Fi measurements coming from an \docAP{} below, causing those particles who are randomly drawn near this \docAPshort{} to be rewarded with a very high weight.
This leads to newly sampled particles in this area and therefore a jump of the estimation.
The situation is resolved after entering another room, which is now shielded by stone walls instead of wooden ones.
Walking down the stairs at \SI{80}{\second} does also recover the localization system using none of the methods.
%
\begin{figure}
\centering
\input{gfx/errorOverTimeWalk0/errorOverTime.tex}
\caption{Error development over time of a single Monte Carlo run of walk 0. Between \SI{10}{\second} and \SI{24}{\second} the Wi-Fi signal was highly attenuated, causing the system to get stuck and producing high errors. Both, the simple and the $D_\text{KL}$ anti-impoverishment method are able to recover early. However, between \SI{65}{\second} and \SI{74}{\second} the simple method produces high errors due to the high random factor involved.}
\label{fig:errorOverTimeWalk0}
\end{figure}
%vielleicht die avg / kde unterscheidung weg lassen? dafür avg, std und 75%
A similar behaviour as described above can be seen in walk 3.
Without a method to recover from impoverishment, the system completely lost track in \SI{xx}{\percent} of the runs and produces high errors.
By using the simple method, the overall error can be reduced and the impoverishment resolved. Nevertheless, unpredictable jumps of the estimation are causing the system to be highly uncertain in some situations, even if those jumps do not last to long.
Only the use of the $D_\text{KL}$ method is able to produce reasonable results.
It is clearly visible that bla outperformce blah at path 4.
Exemplary estimation results for walk 4 can be seen in fig. \ref{}.
The 75 quantil gibt aufschluss, über aufgretenedes impoverishment. due the 100 runs, some of the walks, where impoverishment occured are average out.
neff = 0.85
boxkde 0.2 point2(1,1)
wifi useregionalopt=true
\todo{fuer eins brauchen wir aber noch estimated path}
\todo{neff = 0.85; boxkde 0.2 point2(1,1); wifi useregionalopt=true}
\todo{
BILD: Von einem Pfad der steckenbleibt und den beiden anderen verfahren mit fehler über die zeit.
BILD: WIFI-Fehler unten bei den Kellern.
BILD: Estimation Fehler
}
%To analyse the drawbacks and benefits of the here presented method to resolve sample impoverishment,
@@ -151,6 +169,8 @@ BILD: Estimation Fehler
%probleme mit impoverishment aufzeigen, wo bringt es was, was macht es kaputt etc pp
%%estimation
\subsection{Estimation Methods}
\label{sec:exp:est}
\begin{figure}
\centering
@@ -159,7 +179,7 @@ BILD: Estimation Fehler
\label{fig:realWorldMulti}
\end{figure}
As discussed in chapter \ref{}, the main advantage of a KDE-based estimation is that it provides the "correct" mode of a density, even under a multimodal setting.
As discussed in section \ref{sec:estimation}, the main advantage of a KDE-based estimation is that it provides the "correct" mode of a density, even under a multimodal setting.
A situation in which the system highly benefits from this is illustrated in fig. \ref{fig:realWorldMulti}.
Here, a set of particles splits apart, due to uncertain measurements and multiple possible walking directions.
Indicated by the black dotted line, the resulting bimodal posterior reaches its maximum distance between the modes at \SI{13.4}{\second}.
@@ -180,7 +200,7 @@ Only with new measurements coming from the hallway or other parts of the buildin
\begin{figure}
\centering
\input{gfx/errorOverTime.tex}
\input{gfx/errorOverTimeWalk1/errorOverTime.tex}
\caption{Error development over time of a single Monte Carlo run of the walk calculated between estimation and ground truth. Between \SI{230}{\second} and \SI{290}{\second} to pedestrian was not moving.}
\label{fig:realWorldTime}
\end{figure}