Saba Khalilnaji2018-11-25T23:43:29+00:00http://somesaba.github.ioSaba KhalilnajiMacroTracker 2.02014-11-27T00:00:00+00:00http://somesaba.github.io/posts/macrotracker2<p>I decided to take a break from my <a href="http://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping">SLAM</a> project to give <a href="https://itunesconnect.apple.com/WebObjects/iTunesConnect.woa/ra/ng/app/595564255">MacroTracker</a> some much needed attention. MacroTracker was started immediately after I graduated college. I wanted to start working on side projects and was obsessed with powerlifting and nutrition. I had a powerlifting meet coming up and I wanted to be as lean as possible for my weight class. After expending the effort to track my food and macronutrients (proteins, carbs and fat), I became frustrated with existing nutritional apps. They focused so much on tracking calories and paid very little attention to macros that I felt like they were perpetuating an inappropriate focus on calories. In MacroTracker, I made macronutrients the primary focus.</p>
<p><img src="/images/macrotracker.png" alt="Image 1" /></p>
<h1 id="a-quick-story">A Quick Story</h1>
<p>While working on a group project in college, a girl on my team said she was hungry. I told her we could break for dinner but she explained that she could only eat ~300 more calories that day and didn’t have enough left for a full meal. She then whipped out some nutritional bar and explained that it had only 200 calories and quickly consumed it.</p>
<p>Whatever diet plan this girl is following has put a large focus on calories which has influenced her to think all is well as long as she hits her calories goals. Assuming the girl is eating below her maintenance level, she will most likely lose weight. She’s achieving her goal, so why does it matter if she’s tracking calories or macros? If her only goal is to see the weight on the scale drop then she will likely succeed. However, I think those that are nutritionally concious are either seeking health benefits or aesthetic improvements. In both cases, tracking macros will serve them better.</p>
<h1 id="why-macros">Why Macros</h1>
<p>Not all calories are created equal. 1 gram of protein or carboyhydrates is 4 calories. 1 gram of fat is 9 calories.</p>
<p>If you set goals for your macros, then you are also simultaneously setting caloric goals, and thus if you achieve these goals you will achieve the caloric goals automatically. However, it’s not quite the same the other way around. If I set a caloric goal, there are an infinite number of macronutrient profiles that can achieve this goal. Not all calories are created equal and eating different macronutrient profiles for the same calories can have different results (body composition, energy levels, etc..), which is extremely important to consider for those who care about their health or looks (basically anyone who is tracking calories in the first place). Focusing on calories results in a signifigant loss of information that is detrimental in the long run. People need to explore and learn about what sort of foods and macros work with their bodies and goals especially when considering the long-term maintainability of your diet. Simply tracking calories won’t help you learn.</p>
<h1 id="short-term-macro-tracking-will-improve-long-term-health">Short Term Macro-Tracking Will Improve Long-Term Health</h1>
<p>For those seeking long-term health improvements, your exercise regimine and diet have to be sustainable. Tracking macros or calories is cumbersome and can be stressful. In fact, I would not reccomend using MacroTracker forever. However, it is a powerful educational tool. If you begin to track macros, you will begin to learn what sort of macronutrient profile works for your body and fitness goals. Let’s say you’ve discovered that eating roughly 200g protein, 150g carbs, and 70g fat keep you where you want to be. This is great, but more importantly, you’ve gained knowledge of the macronutrient profiles of different foods. You can now go in a restaurant, look at the menu, and pick something that will fit your target macro profile. You also probably avoid sugary drinks because you have realized that they mess with your macros with little benefit. Overtime, you’ve learned to eat well-rounded meals that you probably had to cook because you discovered the macronutrient profiles of pre-packed or processed foods do not align with your goals.</p>
<p>Tracking macros will help you learn about yourself and the foods you eat while driving you towards a healthier, sustainable lifstyle. If that girl in my class had been tracking her macros, she probably wouldn’t have eaten that bar in lieu of a well-balanced meal.</p>
Supervised Learning Notes2014-06-20T00:00:00+00:00http://somesaba.github.io/posts/supervised-learning-notes<p>I recently started taking <a href="https://www.udacity.com/course/ud675">Georgia Tech’s Machine Learning course</a> to evaluate their program while also getting a nice survey of Machine Learning. I am enjoying the class and wanted to post my notes; the content is structured as it is presented by the professors. I had originally started converting my google doc notes to the format my blog uses (markdown and LaTex) but it proved to be huge waste of time, so I just decided to integrate it below and provide a <a href="https://docs.google.com/document/d/1waHTHvW3zVehVInNMzZW5jiSPUMbjkEEFQoCjGAvwF4/edit?usp=sharing">link</a>. These notes are very high level and primarly serve as a reference. I’ve allowed people to comment on the google doc in case you find any errors :)</p>
<iframe width="100%" height="1000" src="https://docs.google.com/document/d/1waHTHvW3zVehVInNMzZW5jiSPUMbjkEEFQoCjGAvwF4/pub?embedded=true"></iframe>
Robot Foosball Part 3: Artificial Intelligence2014-02-22T00:00:00+00:00http://somesaba.github.io/posts/robot-foosball-part-3--artificial-intelligence<h1 id="introduction">Introduction</h1>
<p>This is the AI part of my foosball project. Be sure to check out <a href="http://somesaba.github.io/posts/robot-foosball-pt1---intro-and-vision/">part 1</a> and <a href="http://somesaba.github.io/posts/robot-foosball-pt2-mechanics-and-hardware/">part 2</a>!</p>
<p>From my (basic) understanding of machine learning there’s the following:</p>
<ul>
<li>Supervised Learning - Given some labeled data, learn a general function to determine the labels for that data without over-fitting. E.g. Classification and Regression</li>
<li>Unsupervised Learning - Given some unlabeled data, find structure in the data. E.g. Clustering and Baum-Welch</li>
<li>Reinforcement Learning - Given some environment, perform actions to observe the consequences. Use these experiences to learn the appropriate actions to maximize expected reward. E.g. Q-Learning and Policy Search</li>
</ul>
<p>The project was my chance to dabble in a real-world application of Reinforcement Learning. I will start by introducing some of the basic ideas behind Reinforcement Learning before talking about my experience in applying it to my foosball robot. The majority of this information is available through <a href="https://www.edx.org/course/uc-berkeleyx/uc-berkeleyx-cs188-1x-artificial-579">Edx’s course on artificial intelligencee</a>, a class I had the privilege of taking in person while I was a student a UC Berkeley. If you find this post interesting, you will do yourself more justice taking the online class since this post almost serves as my own personal cliff notes of one small section of the class. Remember all the code is available on my <a href="http://www.github.com/somesaba/foosball">github</a>.</p>
<h1 id="markov-decision-process">Markov decision process</h1>
<p>I find it fascinating that we can program an agent to learn by experiencing the consequences of its actions in a given environment. Such environments are often modeled by a <a href="http://en.wikipedia.org/wiki/Markov_decision_process">Markov decision process</a> which is defined by the following:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
\text{1. } & \normalsize S \small \text{, a set of states} \\
\text{2. } & \normalsize A \small \text{, a set of actions} \\
\text{3. } & \normalsize P_{a}(s,s')=Pr(s'|s,a) \small \text{, state transition probabilities for a given state and action } \\
\text{4. } & \normalsize R_{a}(s,s') \small \text{, rewards for state transitions for a given state and action } \\
\end{align} %]]></script>
<p><img src="/images/mdp.png" alt="Image 1" /></p>
<p>The image above from <a href="http://en.wikipedia.org/wiki/File:Markov_Decision_Process_example.png">Wikipedia</a> portrays an example of an MDP. MDPs allow us to represent decision problems (what action do I take?) given uncertainty (the nature of our world). It also simplifies things (mostly the math) by allowing us to assume the next state only depends on the current state and action, and not any previous states or actions. With MDPs, we generally want to find a policy. A policy returns an action for every given state. The goal is to find a policy that maximizes expected utility (basically the rewards), this is the optimal policy. I will talk about different ways to determine an optimal policy before introducing Q-learning.</p>
<h1 id="model-based-learning">Model Based Learning</h1>
<p>As you can see in the diagram above, taking an action from a green colored state results in an orange colored “probabilistic state” where the outcome is nondeterministic. There’s a 70% chance that taking action a_0 from state S_1 will result in a +5 reward and end in state S_0, a 20% chance you will end up in state S_2, and a 10% chance that nothing will change. These orange-states are call Q-states and in order for us to write algorithms using MDPs we need to be able to quantify their values. If we think about it intuitively, what would be the value of action a_0 from state S_1? Well, we know there’s a 70% chance of getting a reward of 5, etc…How about .70(value of state S_0 + 5 reward) + .20(value of state S_2) + .10(value of state S_0)? The value of a Q-state (Q-value) can be defined as the weighted sum over all possible transition states of the rewards plus future (discounted) values of the new state, where the weights are the probabilities of each transition:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& Q^{*}(s,a) = \sum_{s'} P_{a}(s,s')[R_{a}(s,s') + \gamma V^{*}(s')] \\
\end{align} %]]></script>
<p>The optimal value of a state (a regular, green state) is therefore the maximum of the Q-values. Basically the value of a state is the best action because Q-values represent state-action pairs:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& V^{*}(s) = max_{a} Q(s,a) \\
& V^{*}(s) = max_{a} \sum_{s'} P_{a}(s,s')[R_{a}(s,s') + \gamma V^{*}(s')] \\
\end{align} %]]></script>
<p>Note: Gamma in these equations can represent greediness. If gamma is 0 the Q-value becomes a weighted sum of the rewards. In our example the Q_value(S_1, a_0) would just be .70(5 reward) if gamma was 0. That means we don’t care if an action gets us killed, as long as we get the largest instantaneous reward between all the other actions. Alternatively, the higher gamma is, the more we consider our future prospects. I keep gamma at .99 for this project.</p>
<p>Now that we can quantify states and actions, we can run an algorithm called <a href="http://en.wikipedia.org">Value Iteration</a> to determine the optimal state values:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
\text{1. } & \forall s\in\normalsize S \small \text{ Set } V_{0}(s) = 0 \\
\text{2. } & \text{Update } V_{k+1}(s) \leftarrow max_{a}\sum_{s'} P_{a}(s,s')[R_{a}(s,s') + \gamma V_{k}(s')] \\
\text{3. } & \text{Repeat until convergence} \\
\end{align} %]]></script>
<p>Value Iteration allows us to determine optimal values as well as the optimal policy for a given state. However, in order to determine the best action given the value of a state, we have to do a bit of work. Calculating the actions from values is called policy extraction and requires you to use the known optimal values, transition probabilities, and rewards to compute the best action. Remember a policy, is just a mapping of states to actions, a table we can use to determine what actions to take given a state.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& \pi^{*}(s) = argmax_{a} \sum_{s'} P_{a}(s,s')[R_{a}(s,s') + \gamma V^{*}(s')] \\
\end{align} %]]></script>
<p>As you can see, determining the optimal policy given state values still requires some work. Another algorithm that could converge faster to the optimal policy is called Policy Iteration:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
\text{1. } & \text{Start with some policy } \pi \text{ and run policy evaluation until the values converge} \\
\text{2. } & \text{Extract a new policy from those values with policy extraction} \\
\text{3. } & \text{Repeat until the policy converges} \\
\end{align} %]]></script>
<p>Policy Evaluation is similar to Value Iteration, except you do not take a max over the available actions because you are limited to your policy:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
\text{1. } & \forall s\in\normalsize S \small \text{ Set } V^{\pi}_{0}(s) = 0 \\
\text{2. } & \text{Update } V^{\pi}_{k+1}(s) \leftarrow \sum_{s'} P_{\pi(s)}(s,s')[R_{\pi(s)}(s,s') + \gamma V^{\pi}_{k}(s')] \\
\text{3. } & \text{Repeat until convergence} \\
\end{align} %]]></script>
<p>The idea here is to pick some policy. Using the diagram above, let’s say we pick a policy of “always takes actions a_0”. Run Policy Evaluation to determine how good that policy was. Now that we’ve gained some information from the environment, we can extract a better policy and repeat. So we have two algorithms we can use to determine the optimal policy, Value Iteration and Policy Iteration.</p>
<h1 id="model-free-learning">Model Free Learning</h1>
<p>Value Iteration and Policy Iteration are great for when you know the all the parameters to your MDP, but what if you don’t? If I apply Value Iteration to foosball, I would have to know all the possible resulting states from hitting the ball in position (200, 100) with my player in position (210, 100), as well as the probabilities for each state. Then I’d need to know this for every possible player-position/ball-position/action combination. And remember, my gamestate is a discretized version of the real world. If my ball is in position (200,100) there’s probably a millimeter or three of real-world tolerance and a whole host of different angles the player could strike the ball with. This would be very difficult to model.</p>
<p>In the general case, it is certainly possible to learn those parameters through machine learning but spending the time learning the parameters seems wasted when that only gets you half way there (you still have to run policy or value iteration).</p>
<p>Fortunately there is a better way! Instead of learning the parameters of the MDP we could try estimating the state values by sampling actions in the actual environment; take an action according to some policy, experience the instantaneous reward, and use that to update your estimate of the state value. This is called temporal difference learning.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& V^{\pi}_{sample}(s) = R_{\pi(s)}(s,s') + \gamma V^{\pi}_{estimate}(s')\\
& V^{\pi}_{estimate}(s) \leftarrow (1-\alpha)V_{estimate}^{\pi}(s) + (\alpha)V^{\pi}_{sample}(s) \\
\end{align} %]]></script>
<p>Alpha is the learning rate, it determines how fast you learn, or how much weight you want to give new experiences. What we’re doing here is calculating a running exponential average for the state value. Overtime this will eventually converge to the correct values for that policy. So now, all we have to do is use policy extraction on our newly estimated state values to improve our policy and repeat until the policy converges. This would essentially be policy iteration where we use estimates for the state values. There’s one problem:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& \pi_{new}(s) = argmax_{a} \sum_{s'} P_{a}(s,s')[R_{a}(s,s') + \gamma V_{estimate}(s')] \\
\end{align} %]]></script>
<p>Remember policy extraction takes a bit of work and it requires us to know the reward and state transition probabilities for the MDP. We’ve taken a model-free approach to learning the state values (for a policy), but we can’t use this knowledge to improve our policy! We can’t use policy extraction unless we also sample transition probabilities and rewards, but we had just decided that was a waste of time…</p>
<p>Let’s go back to the drawing board. Why did we want state values in the first place? Because we’re trying to find the optimal policy. If you look at the equation for policy extraction, it doesn’t really seem to fit well with state values. It almost seems to fit better with…. Q-values! Wait a second…look at the first equation where I define the Q-value for a state.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& \pi^{*}(s) = argmax_{a} \sum_{s'} P_{a}(s,s')[R_{a}(s,s') + \gamma V^{*}(s')] = argmax_{a} Q^{*}(s,a) \\
\end{align} %]]></script>
<p>Policies are just the best action from a given state and Q-values represent actions! So policies should really be extracted from Q-values not state values. What if we learn the Q-values through sampling instead of the state values? What would the sample look like? Well since the value of a state is just the max of the Q-values, it’s no surprise that they are very similar.</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& Q_{sample}(s,a) = R_{a}(s,s') + \gamma max_{a'}Q_{estimate}(s',a') \\
& Q_{estimate}(s,a) \leftarrow (1-\alpha)Q_{estimate}(s,a) + (\alpha)Q_{sample}(s,a) \\
\end{align} %]]></script>
<p>This is Q-learning and this time we get control over which actions to take, where-as with estimated state-values we were limited to a fixed policy (that we couldn’t improve). Q-learning is awesome because it will converge to the optimal Q-values (and therefore policy). Except you have to make sure you explore enough (state and actions) and remember to lower the learning rate (alpha) over time. Since Q-learning will converge to optimal values, it just becomes a question of how fast can I converge which is really a question of exploration vs exploitation (how do I chose which actions to take). That problem can be solved many ways, a simple solution is to take a random action with some probability, epsilon, or otherwise take the best action according to your Q-values. You can also start with some high value for epsilon and decrease it over time. Here’s the basic algorithm for Q-learning:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
\text{1. } & \forall s\in\normalsize S \small \text{ and } a\in\normalsize A \small \text{ set } Q_{estimate}(s,a) = 0 \\
\text{2. } & \text{Take action a from state s} \\
\text{3. } & \text{Recive reward as you transition to a new state, s'} \\
\text{4. } & \text{Update } Q_{estimate}(s,a) \text{ with } Q_{sample}(s,a) \\
\text{5. } & \text{Repeat 2-4 until convergence} \\
\end{align} %]]></script>
<p>There’s still one problem (I promise this is the last one)! Consider the two foosball states below:</p>
<p><img src="/images/foosball_pt3_1.jpg" alt="Image 2" />
<img src="/images/foosball_pt3_2.jpg" alt="Image 3" /></p>
<p>In both cases the agent should probably hit the ball right? Let’s assume the first image is a state we’ve already visited many times and know the best course of action is to hit the ball. The second image however is entirely new and the agent has no idea what to do. The problem here is that we have no way of transferring experiences between similar states. We have to visit every state-action pair combination to learn the optimal values and in foosball that would be:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
& (320*240 \text{ball positions})(40 \text{player y-positions})^{(4 \text{rows})}(5 \text{player angles})^{(4 \text{rows})}
\end{align} %]]></script>
<p>That’s 122 trillion states and that doesn’t even include Q-states. Ain’t nobody got time OR space for that. It would take forever to visit all those states and we probably don’t have enough memory to store all of the Q-values. There must be a better way!</p>
<p>The idea is to mathematically represent states as a linear combination of weights and features, that way we can generalize and transfer experiences between similar states. Features are meant to describe certain characteristics of a state, or in our case, a Q-state, which means we are describing state-action pairs. One feature can represent the possibility of hitting the ball, another feature can represent the possibility of allowing the enemy to score a goal. As the agent learns it adjusts the weights associated with each feature. For example, if the enemy were to score on the agent (which results in a negative reward) while the second feature I mentioned was active, then we would lower the weight for that feature based on the difference between the estimated value for the Q-state and actual (negative) reward we had just received. This is called approximate Q-learning and the algorithm is as follows:</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align}
\text{0. } & \text{Define } Q(s,a) = w_{i}f_{1}(s,a) + w_{2}f_{2}(s,a) + w_{n}f_{n}(s,a) \\
\text{1. } & \forall w\in\normalsize W \small \text{ set } w_{i} = 1 \\
\text{2. } & \text{Take action a from state s} \\
\text{3. } & \text{Recive reward as you transition to a new state, s'} \\
\text{4. } & \text{Update weights } w_{i} \leftarrow w_{i} + \alpha(correction)f_{i}(s,a) \\
\text{5. } & \text{Repeat as needed} \\
\end{align} %]]></script>
<script type="math/tex; mode=display">\begin{align}
correction = (R_{a}(s,s') + \gamma max_{a'}Q(s',a')) - Q(s,a)
\end{align}</script>
<p>When you update the weights in step 4, you are essentially “correcting” them. The correction is the difference between what we thought the Q-value was for the action we took with the actual reward we received plus the best possible (discounted) Q-value for the new state. Those two numbers should be fairly close. You will also notice that we down-weight the correction by alpha, our learning rate and upscale the correction by the actual feature value. The larger (or more prominent) a feature is, the larger it is rewarded for doing good and harsher it is punished for doing bad.</p>
<p>#Implementation</p>
<p>Implementing approximate Q-learning in my foosball robot turned out to be more difficult than I anticipated. I spent days tweaking the rewards, playing around with many different features, until I was able to get the agent to learn something worth showing. The performAction method runs (more or less) every time the vision algorithm updates the gameState if agent is not still doing its thing.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">ApproximateQLearningAgent</span> <span class="kd">extends</span> <span class="n">AbstractFoosballAgent</span> <span class="o">{</span>
<span class="c1">//Weights initialized to 1</span>
<span class="kd">private</span> <span class="n">List</span><span class="o"><</span><span class="n">Float</span><span class="o">></span> <span class="n">featureWeights</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Float</span><span class="o">>(</span><span class="n">Arrays</span><span class="o">.</span><span class="na">asList</span><span class="o">(</span><span class="k">new</span> <span class="n">Float</span><span class="o">[]</span> <span class="o">{</span> <span class="mi">1</span><span class="n">f</span><span class="o">,</span> <span class="mi">1</span><span class="n">f</span><span class="o">,</span> <span class="mi">1</span><span class="n">f</span><span class="o">,</span> <span class="mi">1</span><span class="n">f</span> <span class="o">}));</span>
<span class="c1">// Learning Rate</span>
<span class="kd">private</span> <span class="kt">float</span> <span class="n">alpha</span> <span class="o">=</span> <span class="o">.</span><span class="mi">3</span><span class="n">f</span><span class="o">;</span>
<span class="c1">// Percent Chance for Random Action</span>
<span class="kd">private</span> <span class="kt">float</span> <span class="n">epsilon</span> <span class="o">=</span> <span class="o">.</span><span class="mi">1</span><span class="n">f</span><span class="o">;</span>
<span class="c1">// Discount Factor</span>
<span class="kd">private</span> <span class="kt">float</span> <span class="n">gamma</span> <span class="o">=</span> <span class="o">.</span><span class="mi">99</span><span class="n">f</span><span class="o">;</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">performAction</span><span class="o">()</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">updateWeights</span><span class="o">();</span>
<span class="c1">// Take Action if no one has scored</span>
<span class="k">if</span> <span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getPlayerThatScored</span><span class="o">()</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span>
<span class="k">this</span><span class="o">.</span><span class="na">takeAction</span><span class="o">();</span>
<span class="k">else</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">resetRound</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">takeAction</span><span class="o">()</span> <span class="o">{</span>
<span class="k">if</span><span class="o">(</span><span class="n">Math</span><span class="o">.</span><span class="na">random</span><span class="o">()</span> <span class="o"><</span> <span class="n">epsilon</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">takeRandomAction</span><span class="o">();</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">takeBestAction</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span> </code></pre></figure>
<p>The update weights method is shown below. At this point we are in state s’ and will receive our reward from the previous action. To update the weights, we find the best possible Q-value by iterating over every possible action, and calculating the Q-value with those actions, the current state, and feature weights.</p>
<p>Note: while there are approximately 80 discrete y-positions the bottom player can be in, I’ve reduced this to 40 because there are only 40 possible servo angles and it will significantly cut down the number of possible actions I need to iterate through.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">private</span> <span class="kt">void</span> <span class="nf">updateWeights</span><span class="o">()</span> <span class="o">{</span>
<span class="c1">//Find the maxQValue by calculating the QValue for every action from this state</span>
<span class="kt">float</span> <span class="n">maxQVal</span> <span class="o">=</span> <span class="o">-</span><span class="n">Float</span><span class="o">.</span><span class="na">MAX_VALUE</span><span class="o">;</span>
<span class="c1">// Max over all possible actions (~40,000)</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">rowOneYPos</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">rowOneYPos</span> <span class="o"><</span> <span class="mi">40</span><span class="o">;</span> <span class="n">rowOneYPos</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">rowThreeYPos</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">rowThreeYPos</span> <span class="o"><</span> <span class="mi">40</span><span class="o">;</span> <span class="n">rowThreeYPos</span><span class="o">++)</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">(</span><span class="n">PlayerAngle</span> <span class="n">rowOneAngle</span> <span class="o">:</span> <span class="n">PlayerAngle</span><span class="o">.</span><span class="na">getValues</span><span class="o">())</span> <span class="o">{</span>
<span class="k">for</span> <span class="o">(</span><span class="n">PlayerAngle</span> <span class="n">rowThreeAngle</span> <span class="o">:</span> <span class="n">PlayerAngle</span><span class="o">.</span><span class="na">getValues</span><span class="o">())</span> <span class="o">{</span>
<span class="kt">float</span> <span class="n">qVal</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span>
<span class="n">List</span><span class="o"><</span><span class="n">Float</span><span class="o">></span> <span class="n">featureVals</span> <span class="o">=</span> <span class="k">this</span><span class="o">.</span><span class="na">getFeatureValuesCurrentStateAndActions</span><span class="o">(</span><span class="n">rowOneYPos</span><span class="o">,</span> <span class="n">rowThreeYPos</span><span class="o">,</span> <span class="n">rowOneAngle</span><span class="o">,</span> <span class="n">rowThreeAngle</span><span class="o">);</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">featureVals</span><span class="o">.</span><span class="na">size</span><span class="o">();</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">qVal</span> <span class="o">+=</span> <span class="n">featureWeights</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">i</span><span class="o">)</span> <span class="o">*</span> <span class="n">featureVals</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">i</span><span class="o">);</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">qVal</span> <span class="o">></span> <span class="n">maxQVal</span><span class="o">)</span> <span class="o">{</span>
<span class="n">maxQVal</span> <span class="o">=</span> <span class="n">qVal</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="kt">float</span> <span class="n">diff</span> <span class="o">=</span> <span class="k">this</span><span class="o">.</span><span class="na">getReward</span><span class="o">()</span> <span class="o">+</span> <span class="n">gamma</span> <span class="o">*</span> <span class="n">maxQVal</span> <span class="o">-</span> <span class="n">prevQVal</span><span class="o">;</span>
<span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"UPDATE WEIGHTS diff="</span> <span class="o">+</span> <span class="n">diff</span> <span class="o">+</span> <span class="s">"\tbestQVal="</span> <span class="o">+</span> <span class="n">maxQVal</span> <span class="o">+</span> <span class="s">"\tprevQ"</span> <span class="o">+</span> <span class="n">prevQVal</span> <span class="o">+</span> <span class="s">"\tprevFeatures="</span> <span class="o">+</span> <span class="n">prevFeatureVals</span><span class="o">);</span>
<span class="k">for</span> <span class="o">(</span><span class="kt">int</span> <span class="n">i</span> <span class="o">=</span> <span class="mi">0</span><span class="o">;</span> <span class="n">i</span> <span class="o"><</span> <span class="n">featureWeights</span><span class="o">.</span><span class="na">size</span><span class="o">();</span> <span class="n">i</span><span class="o">++)</span> <span class="o">{</span>
<span class="n">featureWeights</span><span class="o">.</span><span class="na">set</span><span class="o">(</span><span class="n">i</span><span class="o">,</span> <span class="n">featureWeights</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">i</span><span class="o">)</span> <span class="o">+</span> <span class="n">alpha</span> <span class="o">*</span> <span class="n">diff</span> <span class="o">*</span> <span class="n">prevFeatureVals</span><span class="o">.</span><span class="na">get</span><span class="o">(</span><span class="n">i</span><span class="o">));</span>
<span class="o">}</span>
<span class="n">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="s">"UPDATE WEIGHTS New Weights="</span> <span class="o">+</span> <span class="n">featureWeights</span><span class="o">);</span>
<span class="o">}</span></code></pre></figure>
<p>Before I talk about the specific rewards and features used in the demo video I wanted to give some practical advice I learned while getting this to work. The first two are the most important:</p>
<ul>
<li>When you take a non-random action make sure you randomly pick between all actions that had the highest Q-value</li>
<li>Keep your feature values between [0,1] otherwise the update step will cause wild fluctuations in weights. Mine are binary. Put another way, normalize your features.</li>
<li>Remember that feature values are supposed to characterize state-action pairs (repeat this to yourself)</li>
<li>Instantaneous rewards are very important for learning, do not rely on end-state rewards (such as goals)</li>
<li>Your agent will not perform actions that are not featured/rewarded</li>
<li>Start with 1 feature/reward before adding more (baby steps!)</li>
</ul>
<p>I reward the agent for keeping players in line with the ball, for blocking incoming balls, and also hitting the ball forward. Of course there are rewards for scoring a goal (and a negative reward for getting scored on) but those rewards are much less vital to the learning process, at least in my case. The features reflect the rewards; they characterize actions that could potentially block an incoming ball, hit the ball forward if it is within reach, and also try to move the ball in the y-axis if it is directly behind and within reach. As a result those are the only things this agent does.</p>
<h1 id="demo">Demo</h1>
<p>This video below shows some training matches. It is certainly far from perfect, since there are so many actions available to the agent, it can take up to 1 second to calculate the 40,000 Q-values (twice), so I have to hit the ball slow. I wish I had a more powerful computer :)</p>
<center>
<iframe src="//player.vimeo.com/video/87372482" width="720" height="480" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
</center>
<h1 id="conclusions">Conclusions</h1>
<p>My goal with this project was to implement reinforcement learning in a physical environment and get it to work reasonably well. Physical environments are lot harder than simulations due to noise and uncertainty. Nonetheless, I think I’ve accomplished my goal and I have certainly learned a lot. However, the project is not without pitfalls. The agent certainly behaves as expected, but not without noticeable issues, a lot of which can be traced back to noise. Also the large action-space makes it difficult to calculate the next action quickly enough, but a faster computer would easily fix this. If I spend more time improving all aspects of this project (better foosball table, vision algorithm, faster computer, noise reduction), I can probably get the robot to always block an incoming ball, hit the ball forward, etc. But that’s not necessarily how you play foosball, and what happens if the ball gets stuck in the corner (like it did in the video)? Of course I can improve my features, notice how none of my features even consider the enemy position, but I can only go so far with additional features. This is where my model starts to shows its weakness. In foosball, actions are much more complex than just “angle players forward”. Real actions in foosball, are a string of actions in my model. For example, the action “slide the ball over then hit it” or “bounce the ball back and forth then ricochet it off the wall” is what people normally do. If someone was trying to build a truly amazing foosball robot, they would most likely be better off using some physics simulations or treating the game as turn-based game where the offense has a plethora of complex actions to chose from, while the defense just has to block the ball. Maybe you can use Q-learning to train those complex actions! :)</p>
Robot Foosball Part 2: Mechanics and Hardware2014-01-10T00:00:00+00:00http://somesaba.github.io/posts/robot-foosball-pt2-mechanics-and-hardware<h1 id="introduction">Introduction</h1>
<p>I need to warn you that I have a very limited set of tools at my disposal. Namely, a drill, a hand saw, pliers, and screw drivers. One day I will build CAD models of my designs and machine them to perfection, but for now, I need to improvise so don’t judge my handy-work!</p>
<h1 id="parts">Parts</h1>
<p>I already had an Arduino Uno from a different project, so to save money, I decided to reuse it for this one. The big decision now was selecting the right servos to control the players. The most important factor for me was speed. After browsing through all the servos at <a href="http://www.servocity.com/html/servos___accessories.html">servo city</a>, I selected a few based on their speed and price. To my disappointment I had discovered that many servos (especially the fast ones) were limited to 90 degrees of rotation as opposed to 180. Servo city charges $10 to reprogram the servos for 180 degree rotation, or I can buy a programmer myself. Among the three available programmers priced at $25, $50, and $70, I would only be able to use the latter one because the cheaper ones must be connected to a computer running software that’s only available for Windows. Fortunately, I soon discovered that the rotation limits were only applicable to the digital servos so I selected the <a href="http://www.servocity.com/html/hs-225mg_mighty_mini.html">Hitec HS-225MG</a>, an analog servo capable of rotating 60 degrees in 110ms and priced at $25. I followed up that purchase with more aluminum from the hardware store and began the fun!</p>
<h1 id="setup">Setup</h1>
<p>To control the players’ positional movement, I initially thought I’d use a some sort of <a href="http://en.wikipedia.org/wiki/Rack_and_pinion">rack and pinion</a> to transform the servo’s rotational movement to linear movement. After some research, the prices ($50+ each) seemed too high for the size I needed. With some brainstorming, I realized I could accomplish the same thing by simply having an arm extend from the positional servo and attach to the angular servo with a hinge. However, that arm needs to somehow be able to freely change length while remaining rigid in the plane of movement. I drove to the hardware store to see what I could find. I was fortunate enough to find some square aluminum tubing where one size fits perfectly inside another. I cut the tubes to length, drilled the holes for mounting, attached an aluminum L-bracket to the side of foosball table and secured the first servo with its arm.</p>
<p><img src="/images/foosball_pt2_setup_1.jpg" alt="Image 1" />
<img src="/images/foosball_pt2_setup_2.jpg" alt="Image 2" /></p>
<p>For the angular servo I mounted a small brass hinge onto the servo then I used a single bolt and zip tie to secure the other square tube onto the hinge. Next, I experimented with different ways of attaching the servo to the player rod. First, I tried drilling a hole into the plastic end cap that came with the foosball table. Then attached the end cap onto the servo as I would a regular servo arm. The end cap was secured to the rod with duck tape but that is not shown in the image below.</p>
<p><img src="/images/foosball_pt2_setup_4.jpg" alt="Image 3" /></p>
<p>As excepted, this did not work too well. After a few hits the player angles shifted within the end cap and servo neutral was no longer player neutral. Luckily, further experimentation lead to an awesome discovery. The plastic handles that came with the foosball table had a perfectly sized crevice for the circular plastic servo arm. I dissected the plastic handle and mounted the servo to the rod with a screw.</p>
<p><img src="/images/foosball_pt2_setup_5.jpg" alt="Image 4" />
<img src="/images/foosball_pt2_setup_3.jpg" alt="Image 5" /></p>
<p>It wasn’t long before I constructed the mount for the other player rod, attached the arduino onto the table, and connected all the servos to the bread board.</p>
<p><img src="/images/foosball_pt2_setup_6.jpg" alt="Image 6" /></p>
<h1 id="control">Control</h1>
<p>The foosball agent is expected to send servo angle commands to the arduino in 4-byte packets where each byte maps to a particular servo. The arduino simply reads data from the serial port and sets the servo angles.</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="cp">#include <Servo.h>
</span>
<span class="kt">int</span> <span class="n">rowZeroYPositionPin</span> <span class="o">=</span> <span class="mi">3</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">rowZeroAnglePin</span> <span class="o">=</span> <span class="mi">5</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">rowOneYPositionPin</span> <span class="o">=</span> <span class="mi">6</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">rowOneAnglePin</span> <span class="o">=</span> <span class="mi">9</span><span class="p">;</span>
<span class="n">Servo</span> <span class="n">rowZeroYPositionServo</span><span class="p">;</span>
<span class="n">Servo</span> <span class="n">rowZeroAngleServo</span><span class="p">;</span>
<span class="n">Servo</span> <span class="n">rowOneYPositionServo</span><span class="p">;</span>
<span class="n">Servo</span> <span class="n">rowOneAngleServo</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">bytesRead</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">int</span> <span class="n">data</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="kt">void</span> <span class="n">setup</span><span class="p">(){</span>
<span class="c1">//Serial
</span> <span class="n">Serial</span><span class="p">.</span><span class="n">begin</span><span class="p">(</span><span class="mi">9600</span><span class="p">);</span>
<span class="c1">//Attach Servos
</span> <span class="n">rowZeroYPositionServo</span><span class="p">.</span><span class="n">attach</span><span class="p">(</span><span class="n">rowZeroYPositionPin</span><span class="p">);</span>
<span class="n">rowZeroAngleServo</span><span class="p">.</span><span class="n">attach</span><span class="p">(</span><span class="n">rowZeroAnglePin</span><span class="p">);</span>
<span class="n">rowOneYPositionServo</span><span class="p">.</span><span class="n">attach</span><span class="p">(</span><span class="n">rowOneYPositionPin</span><span class="p">);</span>
<span class="n">rowOneAngleServo</span><span class="p">.</span><span class="n">attach</span><span class="p">(</span><span class="n">rowOneAnglePin</span><span class="p">);</span>
<span class="n">rowZeroYPositionServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="mi">90</span><span class="p">);</span>
<span class="n">rowZeroAngleServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="mi">90</span><span class="p">);</span>
<span class="n">rowOneYPositionServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="mi">90</span><span class="p">);</span>
<span class="n">rowOneAngleServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="mi">90</span><span class="p">);</span>
<span class="n">bytesRead</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="kt">void</span> <span class="n">loop</span><span class="p">()</span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="n">Serial</span><span class="p">.</span><span class="n">available</span><span class="p">()</span> <span class="o">>=</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="n">data</span> <span class="o">=</span> <span class="n">Serial</span><span class="p">.</span><span class="n">read</span><span class="p">();</span>
<span class="k">if</span><span class="p">(</span><span class="n">bytesRead</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span> <span class="p">{</span>
<span class="n">rowZeroYPositionServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">data</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="n">bytesRead</span> <span class="o">==</span> <span class="mi">1</span><span class="p">)</span> <span class="p">{</span>
<span class="n">rowZeroAngleServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">data</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="n">bytesRead</span> <span class="o">==</span> <span class="mi">2</span><span class="p">)</span> <span class="p">{</span>
<span class="n">rowOneYPositionServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">data</span><span class="p">);</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="n">rowOneAngleServo</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">data</span><span class="p">);</span>
<span class="p">}</span>
<span class="n">bytesRead</span><span class="o">++</span><span class="p">;</span>
<span class="k">if</span><span class="p">(</span><span class="n">bytesRead</span> <span class="o">==</span> <span class="mi">4</span><span class="p">)</span> <span class="p">{</span>
<span class="n">bytesRead</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">}</span>
<span class="p">}</span></code></pre></figure>
<p>The angular servos are free to move 0 to 180 degrees, however you may realize the positional servos cannot have such freedom. They are in fact limited to about 45 degrees of motion. I manually determined the specific angles for the start and end points of the positional servos and added these offsets to the output of the java program so that the arduino would perform minimal computation. The java foosball agent has to determine an absolute position it wants the player to be in, but then it has to be converted to the proper servo angle. For instance, if the agent decides it wants the player in row 1 to be in position 0 then I have to eventually convert that to a servo angle. If the range of player positions is [0, 240] and the range of servo positions for row 1 is [65, 110], then I must output 65. These conversion are performed in the USBWriter class which wraps the <a href="http://rxtx.qbang.org/wiki/index.php/Main_Page">RxTx Java Library</a> (a library for serial I/O).</p>
<h1 id="demo">Demo</h1>
<p>The following video will demonstrate how the servos control the players.</p>
<center>
<iframe src="//player.vimeo.com/video/84103073" width="720" height="480" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
</center>
<p>The MirrorAgent in the demo will match what the opposing player does.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">MirrorAgent</span> <span class="kd">extends</span> <span class="n">AbstractFoosballAgent</span> <span class="o">{</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">performAction</span><span class="o">()</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getPlayerThatScored</span><span class="o">()</span> <span class="o">==</span> <span class="kc">null</span><span class="o">)</span> <span class="o">{</span>
<span class="n">List</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">intendedYPositions</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Integer</span><span class="o">>(</span><span class="mi">2</span><span class="o">);</span>
<span class="n">List</span><span class="o"><</span><span class="n">PlayerAngle</span><span class="o">></span> <span class="n">intendedPlayerAngles</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">PlayerAngle</span><span class="o">>(</span><span class="mi">2</span><span class="o">);</span>
<span class="n">intendedPlayerAngles</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getPlayerAngleForRow</span><span class="o">(</span><span class="mi">0</span><span class="o">));</span>
<span class="n">intendedPlayerAngles</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getPlayerAngleForRow</span><span class="o">(</span><span class="mi">2</span><span class="o">));</span>
<span class="n">intendedYPositions</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getRowYPosition</span><span class="o">(</span><span class="mi">0</span><span class="o">));</span>
<span class="n">intendedYPositions</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getRowYPosition</span><span class="o">(</span><span class="mi">2</span><span class="o">));</span>
<span class="n">usbWriter</span><span class="o">.</span><span class="na">setPlayerPositions</span><span class="o">(</span><span class="n">intendedYPositions</span><span class="o">,</span> <span class="n">intendedPlayerAngles</span><span class="o">);</span>
<span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">resetRound</span><span class="o">();</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<center>
<iframe src="//player.vimeo.com/video/83951632" width="720" height="480" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
</center>
Robot Foosball Part 1: Intro and Vision2014-01-03T00:00:00+00:00http://somesaba.github.io/posts/robot-foosball-pt1---intro-and-vision<h1 id="introduction">Introduction</h1>
<p>Months ago, while burning the midnight oil at work for a late-night release, my team and I would take short foosball breaks when we had to wait for our servers to warm their caches. After one such escapade, my boss mentioned how cool it would be to build a foosball playing robot for the company’s next hackathon. The fatigue from the day instantly faded. My persistent desire to build and create had been unsatisfied for far too long. I produced <a href="http://macrotrackerapp.com">MacroTracker</a> earlier in the year, but this time I wanted to build something tangible and I wanted it to have AI. Autonomous foosball was the perfect answer!</p>
<p>I broke the project down into 3 milestones. First was the vision aspect, convert the images captured by the camera to some grid-based gamestate object and display it next to the raw camera feed. Second, I needed to complete the hardware and mechanics to control the players. Finally, I needed to write the AI to control the players, specifically I wanted it to learn how to play on its own with Q-learning or perhaps some sort of policy-search. All the code is available on my <a href="https://github.com/somesaba/foosball">github</a>.</p>
<h1 id="parts">Parts</h1>
<p>I bought a <a href="http://www.amazon.com/Mini-Table-Top-Foosball-Everything/dp/B003YHD1DE">$20 table-top foosball</a> table from Amazon to start. Then, I needed a camera capable of tracking a ball at high speeds. I knew from experience that webcams were poor choices as they tend to be anywhere from 15-30fps, usually on the lower side. I arbitrarily decided I wanted a camera capable of 100fps and began my search. Unfortunately, I quickly discovered that high speed cameras are prohibitively expensive ($500+) and a lot of them use firewire, cameralink, and ethernet ports for data transfer. I wanted to limit myself to USB and eventually found myself with several options:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. Point Grey Flea3 960p @ 150fps $700+
2. GoPro Black Edition 960p @ 100fps $400
3. Highend Android Phone 320p @ 120fps $200-800
4. PS3 Eye Camera 320p @ 120fps $20
</code></pre></div></div>
<p>The Point Grey camera was too expensive for this project (but awesome). I considered the GoPro for a few minutes as it would have been useful for other purposes, but it turns out the only [reasonable] way to stream video is over WiFi which results in a delay that makes it unsuitable for vision applications. Since it was so cheap, I ended up purchasing the PS3 Eye camera to begin coding. I hit a road bump when I discovered the camera was only supported by Sony on Windows and since I had OS X, there was a problem. <a href="http://webcam-osx.sourceforge.net/">Macam</a>, an OS X driver for usb webcams, supported my camera, but was unreliable and often crashed. After digging through blogs and forum posts, I learned that some people had managed to attain 60fps in Ubunutu with special drivers. I decided to dual boot my macbook air with Ubunutu and dove into coding. Since, I was disappointed with the framerate I was working with, I continued to dig into other cameras on the side. The HTC One and Galaxy S4 were both capable of capturing high speed video. Since I was in the market for a new phone anyway, I considered this option. Unfortunately, after some testing, I discovered the Camera API for Android was generally capped at 15-30fps (depending on the device) when supplying live camera data. I settled with what I had and trudged on!</p>
<h1 id="setup">Setup</h1>
<p>After purchasing some aluminum from my local hardware store, I bent it to the appropriate shape and marked the holes for the camera. To avoid making extra holes in the camera casing, I decided to reuse two of the holes already used to hold it together. Once the camera was secured to the aluminum, I marked the holes on the foosball table itself and attached the whole thing.</p>
<p><img src="/images/foosball_pt1_setup_1.jpg" alt="Image 1" /><img src="/images/foosball_pt1_setup_2.jpg" alt="Image 2" />
<img src="/images/foosball_pt1_setup_3.jpg" alt="Image 3" /><img src="/images/foosball_pt1_setup_4.jpg" alt="Image 4" /></p>
<p>Eventually, I discovered that the speed of the camera and the RGB values of certain pixels differed dramatically as the lighting changed. My initial tracking algorithms would behave slightly differently at different times of the day, in different parts of the room, and so on. There was only so much I could do on the software side without diving into more advanced methods of image processing, which was something I was not interested in doing for this project. The easy solution was to provide static lighting for the camera so the framerate and pixels values remained constant. Two 120V 5W lightbulbs were duck-taped to the aluminum stands with gray construction paper used as shielding for my eyes.</p>
<p><img src="/images/foosball_pt1_setup_5.jpg" alt="Image 5" /></p>
<p>I cheated even more. If you compare the image above with the image of the original foosball table in the Amazon link, you will notice many differences. All the changes I made reduce noise my vision algorithm:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. Sharpied the white lines black to reduce white pixels in frame
2. Painted the black-and-white soccer ball all white
3. Sharpied the metal handles black to reduce white pixels in frame
4. Sharpied the red-and-white players all red
5. Sharpied the yellow-and-blue players all blue
6. Sharpied goalies black to distinguish from outer players
7. Cut off heads of bottom players to eliminate black pixels near player-tracking areas
</code></pre></div></div>
<h1 id="model">Model</h1>
<p>The state of a foosball game can be modeled by a 2D grid with position data for the players and the balls. The ball is free to move anywhere on this grid so its position can be represented by an x,y coordinate. The players however are limited to one axis, on which they are free to rotate or slide (in my case the y-axis). Furthermore, the players are also fixed in number as well as relative distances between each other. This nature allows me to divide the gameState into static and dynamic properties. The static properties are the size of the 2D grid, the number of players per row, the distance between players for each row, and finally the x-coordinates for each row. The size of gameState was set to 320x240 (same as the output of the webcam) and the other static properties were determined manually afterwards. The dynamic properties are essentially the x,y-coordinates of the ball, the y-coordinates of one player (in my case the bottom player) for each row, as well the the player angle for each row. (Note: My model is missing a list of controllable rows). I further discretized the PlayerAngle into an enum with 5 possible states (see below). The horizontal angles allow the ball to pass underneath whereas the others do not. Lastly, the playerTheScored property is used to determine if the game is over and if so, who had scored, it remains null while the game is live.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">GameState</span> <span class="o">{</span>
<span class="c1">//Static State Variables</span>
<span class="c1">// The number of discrete points in the x-axis, perpendicular to the players</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">xLength</span><span class="o">;</span>
<span class="c1">// The number of discrete points in the y-axis, parallel to the players</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="kt">int</span> <span class="n">yLength</span><span class="o">;</span>
<span class="c1">// A Map of the number of the players each row of players has</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="n">Map</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">rowToPlayerCountMap</span><span class="o">;</span>
<span class="c1">// A Map of the distance between players for each row</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="n">Map</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">rowToPlayerDistanceMap</span><span class="o">;</span>
<span class="c1">// A Map of the fixed x-position of the players for each row</span>
<span class="kd">private</span> <span class="kd">final</span> <span class="n">Map</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">rowToXPositionMap</span><span class="o">;</span>
<span class="c1">//Dynamic State Variables</span>
<span class="kd">private</span> <span class="kd">volatile</span> <span class="kt">int</span> <span class="n">prevBallXPosition</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">volatile</span> <span class="kt">int</span> <span class="n">prevBallYPosition</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">volatile</span> <span class="kt">int</span> <span class="n">ballXPosition</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">volatile</span> <span class="kt">int</span> <span class="n">ballYPosition</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">ConcurrentMap</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">Integer</span><span class="o">></span> <span class="n">rowToYPositionMap</span><span class="o">;</span>
<span class="kd">private</span> <span class="n">ConcurrentMap</span><span class="o"><</span><span class="n">Integer</span><span class="o">,</span> <span class="n">PlayerAngle</span><span class="o">></span> <span class="n">rowToAngleMap</span><span class="o">;</span>
<span class="kd">private</span> <span class="kd">volatile</span> <span class="n">Player</span> <span class="n">playerThatScored</span> <span class="o">=</span> <span class="kc">null</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">enum</span> <span class="n">PlayerAngle</span> <span class="o">{</span>
<span class="n">FORWARD_HORIZONTAL</span><span class="o">,</span> <span class="n">FORWARD_ANGLED</span><span class="o">,</span> <span class="n">VERTICAL</span><span class="o">,</span> <span class="n">BACKWARD_ANGLED</span><span class="o">,</span> <span class="n">BACKWARD_HORIZONTAL</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kd">enum</span> <span class="n">Player</span> <span class="o">{</span>
<span class="n">SELF</span><span class="o">,</span> <span class="n">ENEMY</span><span class="o">;</span>
<span class="o">}</span></code></pre></figure>
<p>I need to preface the tracking algorithms with a quick explanation of the PotenialPositionRectangle object. It essentially represents a rectangle but spawns from a single point (it becomes the 2x2 rectangle around its original point). It has methods for determining whether another point is a member of the rectangle (within its bounds) or nearby (within a given distance). Furthermore, it has methods for adding new points, which will expand its own bounds to engulf the new point. Lastly, all the methods above can also be performed with another PotenialPositionRectangle; you can determine whether the two rectangles overlap, are within a certain distance from each other, and merge one into the other.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="kd">public</span> <span class="kd">class</span> <span class="nc">PotentialPositionRectangle</span> <span class="o">{</span>
<span class="kd">private</span> <span class="kt">int</span> <span class="n">xStart</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">int</span> <span class="n">xEnd</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">int</span> <span class="n">yStart</span><span class="o">;</span>
<span class="kd">private</span> <span class="kt">int</span> <span class="n">yEnd</span><span class="o">;</span>
<span class="kd">public</span> <span class="nf">PotentialPositionRectangle</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span><span class="o">,</span> <span class="kt">int</span> <span class="n">y</span><span class="o">)</span> <span class="o">{</span>
<span class="n">xStart</span> <span class="o">=</span> <span class="n">x</span> <span class="o">-</span> <span class="mi">1</span><span class="o">;</span>
<span class="n">xEnd</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="o">;</span>
<span class="n">yStart</span> <span class="o">=</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">;</span>
<span class="n">yEnd</span> <span class="o">=</span> <span class="n">y</span> <span class="o">+</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isMember</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span><span class="o">,</span> <span class="kt">int</span> <span class="n">y</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">x</span> <span class="o">>=</span> <span class="n">xStart</span> <span class="o">&&</span> <span class="n">y</span> <span class="o">>=</span> <span class="n">yStart</span> <span class="o">&&</span> <span class="n">x</span> <span class="o"><=</span> <span class="n">xEnd</span> <span class="o">&&</span> <span class="n">y</span> <span class="o"><=</span> <span class="n">yEnd</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isWithin</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span><span class="o">,</span> <span class="kt">int</span> <span class="n">y</span><span class="o">,</span> <span class="kt">int</span> <span class="n">within</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="n">x</span> <span class="o">>=</span> <span class="n">xStart</span> <span class="o">-</span> <span class="n">within</span> <span class="o">&&</span> <span class="n">y</span> <span class="o">>=</span> <span class="n">yStart</span> <span class="o">-</span> <span class="n">within</span> <span class="o">&&</span> <span class="n">x</span> <span class="o"><=</span> <span class="n">xEnd</span> <span class="o">+</span> <span class="n">within</span> <span class="o">&&</span> <span class="n">y</span> <span class="o"><=</span> <span class="n">yEnd</span> <span class="o">+</span> <span class="n">within</span><span class="o">;</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">doesCollide</span><span class="o">(</span><span class="n">PotentialPositionRectangle</span> <span class="n">rect</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxStart</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyStart</span><span class="o">())</span>
<span class="o">||</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxStart</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyEnd</span><span class="o">())</span>
<span class="o">||</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxEnd</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyStart</span><span class="o">())</span>
<span class="o">||</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxEnd</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyEnd</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">boolean</span> <span class="nf">isWithin</span><span class="o">(</span><span class="n">PotentialPositionRectangle</span> <span class="n">rect</span><span class="o">,</span> <span class="kt">int</span> <span class="n">within</span><span class="o">)</span> <span class="o">{</span>
<span class="k">return</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxStart</span><span class="o">()</span> <span class="o">-</span> <span class="n">within</span><span class="o">,</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyStart</span><span class="o">()</span> <span class="o">-</span> <span class="n">within</span><span class="o">)</span>
<span class="o">||</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxStart</span><span class="o">()</span> <span class="o">-</span> <span class="n">within</span><span class="o">,</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyEnd</span><span class="o">()</span> <span class="o">+</span> <span class="n">within</span><span class="o">)</span>
<span class="o">||</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxEnd</span><span class="o">()</span> <span class="o">+</span> <span class="n">within</span><span class="o">,</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyStart</span><span class="o">()</span> <span class="o">-</span> <span class="n">within</span><span class="o">)</span>
<span class="o">||</span> <span class="k">this</span><span class="o">.</span><span class="na">isPotentialMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxEnd</span><span class="o">()</span> <span class="o">+</span> <span class="n">within</span><span class="o">,</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyEnd</span><span class="o">()</span> <span class="o">+</span> <span class="n">within</span><span class="o">);</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">addRectangle</span><span class="o">(</span><span class="n">PotentialPositionRectangle</span> <span class="n">rect</span><span class="o">)</span> <span class="o">{</span>
<span class="k">this</span><span class="o">.</span><span class="na">addMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxStart</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyStart</span><span class="o">());</span>
<span class="k">this</span><span class="o">.</span><span class="na">addMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxStart</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyEnd</span><span class="o">());</span>
<span class="k">this</span><span class="o">.</span><span class="na">addMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxEnd</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyStart</span><span class="o">());</span>
<span class="k">this</span><span class="o">.</span><span class="na">addMember</span><span class="o">(</span><span class="n">rect</span><span class="o">.</span><span class="na">getxEnd</span><span class="o">(),</span> <span class="n">rect</span><span class="o">.</span><span class="na">getyEnd</span><span class="o">());</span>
<span class="o">}</span>
<span class="kd">public</span> <span class="kt">void</span> <span class="nf">addMember</span><span class="o">(</span><span class="kt">int</span> <span class="n">x</span><span class="o">,</span> <span class="kt">int</span> <span class="n">y</span><span class="o">)</span> <span class="o">{</span>
<span class="k">if</span> <span class="o">(</span><span class="n">x</span> <span class="o"><=</span> <span class="n">xStart</span><span class="o">)</span> <span class="o">{</span>
<span class="n">xStart</span> <span class="o">=</span> <span class="n">x</span> <span class="o">-</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">y</span> <span class="o"><=</span> <span class="n">yStart</span><span class="o">)</span> <span class="o">{</span>
<span class="n">yStart</span> <span class="o">=</span> <span class="n">y</span> <span class="o">-</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">x</span> <span class="o">>=</span> <span class="n">xEnd</span><span class="o">)</span> <span class="o">{</span>
<span class="n">xEnd</span> <span class="o">=</span> <span class="n">x</span> <span class="o">+</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="k">if</span> <span class="o">(</span><span class="n">y</span> <span class="o">>=</span> <span class="n">yEnd</span><span class="o">)</span> <span class="o">{</span>
<span class="n">yEnd</span> <span class="o">=</span> <span class="n">y</span> <span class="o">+</span> <span class="mi">1</span><span class="o">;</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="o">}</span></code></pre></figure>
<h1 id="player-tracking">Player Tracking</h1>
<p>Below you will see a glimpse of the game state visualization (sorry for the horrible picture). Notice the four white rectangles that span across every row? Those rectangles are drawn using the GameState field rowToXPositionMap (the x position from the map is the center of the rectangles that are 10 pixels wide). The player tracking algorithm limits itself to the x-bounds of that rectangle. Starting from y/2 (remember I only need to find the y-position and PlayerAngle of the bottom player), I start looking for blue-like or red-like pixels, depending on the row, until I reach the bottom of image. If I find a pixel matching my color thresholds, I iterate over a list of PotentialPositionRectangles and add them to the rectangles that I am with N pixels of. You can play with N, mine is 0. If I find myself without any memberships, I create a new PotentialPositionRectangle with the current point and add it to the list, but if i find myself with more than one membership then the PotentialPositionRectangles that I am a member of might overlap so I merge the ones that do and remove the extras from the list. What this results in is a list rectangles that encapsulate my pixels, it’s not quite the contiguous pixels of my choice, but it’s similar. Next, I iterate over this list and remove any rectangles that are less than 15pixels in height, I call these “noisy rectangles”. Usually this only leaves one remaining rectangle, but in the case there’s multiple I pick the one closest to the current y-position of that player. If you look closely at the picture below, you will see red or blue rectangles around the bottom players with some “noisy rectangles” above or below.</p>
<p>Player angles are then determined from y-positions we just found. Notice the red and blue rectangles to the left and right of players in the image below. Those are the thresholds for when a player is in a forward or backward type position. I scan the pixels in those boxes and if I find more than M pixels in the box then we know we’re in a non-vertical position (M is set to 25 in my case). From here, we scan pixels in another box of the same size immediately adjacent to drawn boxes to determine if we are in a horizontal position or not.</p>
<p><img src="/images/foosball_pt1_tracking_1.png" alt="Image 5" /></p>
<h1 id="ball-tracking">Ball Tracking</h1>
<p>My first pass at the ball tracking algorithm was to use the average position of all the white-like pixels. After that failed miserably, I came up with the algorithm I explained above. After looping over the entire image, I had my list of PotentialPositionRectangles (of white pixels) with many different candidates. I then loop over the PotentialPositionRectangles and scored them based on their distance away from being a perfect 22x22 square. The lowest score became the ball’s new position. This worked fairly well except occasionally a noisy PotentialPositionRectangle would score well causing the ball to jump to the other side of the table then jump back. I solved this by including the distance from the previous ball position in the score. It’s still not perfect but works fairly well.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="k">for</span> <span class="o">(</span><span class="n">PotentialPositionRectangle</span> <span class="n">rect</span> <span class="o">:</span> <span class="n">potentialBallPositionRectangles</span><span class="o">)</span> <span class="o">{</span>
<span class="kt">double</span> <span class="n">score</span> <span class="o">=</span> <span class="n">Math</span><span class="o">.</span><span class="na">sqrt</span><span class="o">(</span><span class="n">Math</span><span class="o">.</span><span class="na">pow</span><span class="o">(</span><span class="mi">22</span> <span class="o">-</span> <span class="n">rect</span><span class="o">.</span><span class="na">getXLen</span><span class="o">(),</span> <span class="mi">2</span><span class="o">)</span>
<span class="o">+</span> <span class="n">Math</span><span class="o">.</span><span class="na">pow</span><span class="o">(</span><span class="mi">22</span> <span class="o">-</span> <span class="n">rect</span><span class="o">.</span><span class="na">getYLen</span><span class="o">(),</span> <span class="mi">2</span><span class="o">))</span>
<span class="o">+</span> <span class="o">(</span><span class="n">Math</span><span class="o">.</span><span class="na">sqrt</span><span class="o">(</span><span class="n">Math</span><span class="o">.</span><span class="na">pow</span><span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getBallXPosition</span><span class="o">()</span> <span class="o">-</span> <span class="n">xPosition</span><span class="o">,</span> <span class="mi">2</span><span class="o">)</span>
<span class="o">+</span> <span class="n">Math</span><span class="o">.</span><span class="na">pow</span><span class="o">(</span><span class="n">gameState</span><span class="o">.</span><span class="na">getBallYPosition</span><span class="o">()</span> <span class="o">-</span> <span class="n">yPosition</span><span class="o">,</span> <span class="mi">2</span><span class="o">)))</span> <span class="o">/</span> <span class="mi">3</span><span class="o">;</span>
<span class="o">.</span>
<span class="o">.</span>
<span class="o">.</span>
<span class="o">}</span></code></pre></figure>
<h1 id="optimizations">Optimizations</h1>
<p>The basic loop in the gameStateUpdater originally did the following:</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="k">while</span> <span class="o">(!</span><span class="n">isShutdown</span><span class="o">)</span> <span class="o">{</span>
<span class="n">BufferedImage</span> <span class="n">img</span> <span class="o">=</span> <span class="n">stateReader</span><span class="o">.</span><span class="na">readState</span><span class="o">();</span>
<span class="c1">// Update Players positions in current thread (~10ms)</span>
<span class="k">this</span><span class="o">.</span><span class="na">updatePlayerPositions</span><span class="o">(</span><span class="n">img</span><span class="o">);</span>
<span class="c1">// Update Ball Position in current thread (~40-50ms)</span>
<span class="k">this</span><span class="o">.</span><span class="na">updateBallPosition</span><span class="o">(</span><span class="n">img</span><span class="o">);</span>
<span class="n">notifyListeners</span><span class="o">();</span>
<span class="o">}</span></code></pre></figure>
<p>The whole thing took on average 55ms which made visualization seem a bit slow. I decided to multi-thread the the ball position calculation by splitting it up into 5 regions where the divider was the player bar. Each region would calculate its own lowest scoring PotentialPositionRectangle and in the end it picks the best of the 5. This is possible because the ball can’t be seen when its directly under the bar and the original algorithm would have chosen the side where the ball was sticking out the most anyway, so there’s little difference between the results of the two. The new code calculates the player and ball position within ~15ms and the visualization is much more fluid.</p>
<figure class="highlight"><pre><code class="language-java" data-lang="java"><span class="k">while</span> <span class="o">(!</span><span class="n">isShutdown</span><span class="o">)</span> <span class="o">{</span>
<span class="n">BufferedImage</span> <span class="n">img</span> <span class="o">=</span> <span class="n">stateReader</span><span class="o">.</span><span class="na">readState</span><span class="o">();</span>
<span class="c1">// Start Calculating Ball Positions</span>
<span class="n">List</span><span class="o"><</span><span class="n">Future</span><span class="o"><</span><span class="n">PotentialPositionRectangle</span><span class="o">>></span> <span class="n">potentialBallPositions</span> <span class="o">=</span> <span class="k">this</span><span class="o">.</span><span class="na">startCalculatingBallPositionsAsync</span><span class="o">(</span><span class="n">img</span><span class="o">);</span>
<span class="c1">// Update Player positions in current thread</span>
<span class="k">this</span><span class="o">.</span><span class="na">updatePlayerPositions</span><span class="o">(</span><span class="n">img</span><span class="o">);</span>
<span class="c1">// Wait for futures and update ball position</span>
<span class="k">this</span><span class="o">.</span><span class="na">updateBallPosition</span><span class="o">(</span><span class="n">potentialBallPositions</span><span class="o">);</span>
<span class="n">notifyListeners</span><span class="o">();</span>
<span class="o">}</span></code></pre></figure>
<h1 id="demo">Demo</h1>
<center>
<iframe src="//player.vimeo.com/video/83826553" width="720" height="480" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
</center>
Hello World2013-05-30T00:00:00+00:00http://somesaba.github.io/posts/hello-world<p>Hello everyone,</p>
<p>I have decided to start blogging and this is my first post! My blog is hosted on <a href="http://pages.github.com/">GitHub Pages</a> with the <a href="https://github.com/caarlos0/up">UP theme</a> for jekyll. My plan is to primarily post about my robotics/technology projects and my nutrition/fitness endeavors. Stay tuned for posts on an autonomous foosball-playing-robot that uses <a href="http://webdocs.cs.ualberta.ca/~sutton/book/ebook/the-book.html">reinforcement learning</a>, how I plan to train/recover from my L4/L5 herniated disc, and my simple <a href="http://macrotrackerapp.com">iphone app</a> for tracking macronutrients.</p>